Apple researchers have shown a humanoid robot, a robot built in a human shape and walking on two legs, doing everyday moves in real homes after learning them mostly from practice runs made by computer. In a paper posted on Nov. 4, eight researchers, all listed under Apple, describe a system they call HALO. They say it never saw a real home during training, yet succeeded in 91% of 360 tries at single tasks in six real homes it had never seen.
Skills a robot learns in a computer simulation have often failed to carry over to a real machine. Some teams instead collect training data by having a person steer the robot remotely, as with a remote control, or wear motion-capture gear. The researchers say that takes at least an hour of a person's time for every hour of data.
HALO's practice runs instead come from 1,000 rooms generated by a computer program, such as living rooms, bedrooms and offices. Software made up most of the robot's movements, placed them in those rooms and showed the robot what its own camera would see there. The researchers say the amount of practice grows with computing power rather than people's time, and they made it on 250 powerful AI chips. Some recordings of real people moving were still used, for example to teach the robot how to push a box, but no one steered the robot to collect the practice runs.
The robot is a Unitree G1, fitted with an extra camera on its head and a Mac mini computer in a box on its back, and the system runs on the robot itself, though the planning step can call an AI model over the internet. A person can ask out loud for things such as "Push box away", "Follow me" or "Sit inside". An off-the-shelf AI model turns each request into steps about every two seconds. HALO's own motion model then plans about five seconds of whole-body movement for each step, about three times a second.
In the tests, the robot walked to named objects, sat on chairs, stepped sideways through half-open doors, crouched and crawled under tables, and pushed boxes. A try counted as a success only if the robot finished without falling or getting stuck. When the researchers chained two or three tasks in one real room, the whole chain succeeded 70% of the time on average, from half the time for the hardest chain to every time for the easiest.
The researchers say one cause of failure was a task ending with the next target out of the camera's view. In the hardest chain, three of the five failed tries ended a crouch with the robot's head turned away from the box it was meant to push next. The results come from the researchers' own tests. Apple posted the code for making the data, training and running the system on GitHub on Nov. 5.
The researchers also list limits. Each new skill still needs a person to write its description and set its key poses, so the list of skills grows only as fast as people can define them. The robot relies mostly on its cameras and cannot feel when it touches something, so it loses track of an object once it leaves the camera's view. The researchers expect to need a more capable planner as the list of tasks grows.