{"id":54794,"date":"2025-05-21T23:38:27","date_gmt":"2025-05-21T21:38:27","guid":{"rendered":"https:\/\/www.apfelpatient.de\/?p=54794"},"modified":"2025-05-21T23:38:29","modified_gmt":"2025-05-21T21:38:29","slug":"apple-uses-i-perspective-to-train-robots","status":"publish","type":"post","link":"https:\/\/www.apfelpatient.de\/en\/news\/apple-nutzt-ich-perspektive-zur-schulung-von-robotern","title":{"rendered":"Apple uses first-person perspective to train robots"},"content":{"rendered":"

Apple has published a new study demonstrating how humanoid robots can be trained more efficiently in the future. What's unique about it is that the machines no longer just learn from other robots, but directly from humans\u2014from their perspective. This is made possible by an Apple Vision Pro, which is used for data collection. The study is called "Humanoid Policy \u223c Human Policy" and was conducted in collaboration with MIT, Carnegie Mellon, the University of Washington, and UC San Diego.<\/strong><\/p>\n\n\n\n

Robot training has so far been complex, expensive, and difficult to scale. Humans usually have to remotely control robots via complex teleoperation systems to teach them specific tasks. This takes time and resources. Apple is now taking a different approach: Robots are designed to learn human actions directly from videos filmed from a first-person perspective. The test person wears a headset\u2014such as the Apple Vision Pro\u2014and performs simple or complex actions with objects. The recordings are then used to train an AI that can instruct humanoid robots.<\/p>\n\n\n\n

Ego shots instead of expensive robot data<\/h4>\n\n\n\n

The study investigated how human demonstrations from the first-person perspective (i.e., directly from the point of view of the person performing the action) can be used as training data. A total of over 25,000 human and 1,500 robot demonstrations were collected. This data was combined into a common format that the researchers call "PH2D." The goal was to develop a single AI policy that works based on both data sources (human and robot). The researchers emphasize that learning solely from robot data is inefficient. Acquisition is expensive, requires complex equipment, and is difficult to implement on a large scale. Egocentric human demonstrations, on the other hand, create a scalable alternative that can be used more flexibly.<\/p>\n\n\n\n

How the Apple Vision Pro was used<\/h4>\n\n\n\n

To capture this type of training data, the researchers developed a special app for the Apple Vision Pro. It uses the headset's bottom left camera and Apple's ARKit framework to capture head and hand movements in three dimensions. This allows for first-person capture of actions\u2014including all the motion data a robot needs to imitate the action.<\/p>\n\n\n\n

\"\"
Image: Apple<\/figcaption><\/figure>\n\n\n\n

To make the method accessible to more people, a mount was also developed that allows a ZED Mini stereo camera to be attached to other headsets such as the Meta Quest 3. This allows similar 3D motion data to be captured \u2013 only at a significantly lower cost.<\/p>\n\n\n\n

Faster and cheaper training<\/h4>\n\n\n\n

Another advantage: The training data can be collected very quickly. While traditional teleoperations often take a long time, the new method allows a complete demonstration to be recorded in just a few seconds. This not only reduces costs but also makes training much easier to scale. A side note: Since humans move much faster than robots, the recorded videos had to be artificially slowed down during the training process \u2013 by a factor of four. This allows the robots to follow human actions without the need for further adjustments.<\/p>\n\n\n\n