Having learned a lot from sharing my previous rock climbing demos, I realized that a lot of rock climbing analysis is well-suited for 3D. Even something as simple as supporting videos where the person filming moves with the climber requires 3D information.
To get the depth information, I used my iPhone 15 Pro's LiDAR depth sensor through my local iPhone app. I recorded the video and depth measurements from my app, and I ran the rest of the analysis on my computer. I used ViTPose+ Large for pose estimation and SAM 3.1 to segment the holds, both models accessed through the VLM Run Gateway.
I think the holds activation is better, and I like the final view of all of the holds in 3D. It's also interesting to see the distance traveled in meters. Plus, it looks cool and it feels like a video game.
To get the depth information, I used my iPhone 15 Pro's LiDAR depth sensor through my local iPhone app. I recorded the video and depth measurements from my app, and I ran the rest of the analysis on my computer. I used ViTPose+ Large for pose estimation and SAM 3.1 to segment the holds, both models accessed through the VLM Run Gateway.
I think the holds activation is better, and I like the final view of all of the holds in 3D. It's also interesting to see the distance traveled in meters. Plus, it looks cool and it feels like a video game.
Let me know what you think!
The analysis code is open-source on GitHub: https://github.com/jeremyipark/vision-demos
nice
Is the LiDAR sufficiently accurate at those distances?