Researchers at Tohoku University in Japan and the University of Hong Kong have built an AI called DynaConTalk that turns a recording of someone speaking into the movements of a talking 3D character. It produces shifts in posture, hand gestures and facial expressions that follow the words. Tohoku described the work in an English announcement on Nov. 13, and says it could be used to animate characters in video games and films, virtual presenters and online lessons.
People talk with their whole bodies, and that is hard for software to copy. Tohoku says big changes in posture can drown out small ones, such as a finger gesture or a change of expression, so AI systems struggle to learn them. The result can be the same gestures over and over, or unnatural shaking. DynaConTalk splits movement into slower and faster parts and learns them together, which Tohoku compares to separating the low and high notes in a sound. That lets it catch both broad changes in posture and small, brief gestures. DynaConTalk also looks back at the moves it has just made before making the next ones, so a long speech has fewer sudden jumps.
Animators can steer it. Tohoku says a creator could set a character's starting pose or give it a sample movement, then let the AI fill in the gestures and expressions around it.
In the researchers' own test, 30 people compared clips from DynaConTalk and two earlier systems from other teams, without knowing which was which. On average they rated DynaConTalk's clips highest for looking real, matching what was said, keeping time with the speech and variety of movement.
The researchers first posted the paper and code online in early October. The code, an app for making and editing the movement and the trained AI are all online to download, though they run best on a computer with an NVIDIA graphics card, take many minutes without one, and need a separately licensed 3D body model.
The team hopes to extend it to more whole-body movement, such as walking, but says the recordings available today hold relatively few examples of big movements, and speech alone doesn't tell a character where to go. The AI learned from recordings of people speaking English, and the researchers say how well it handles a wider range of speakers is still to be tested. The work was published in the journal ACM Transactions on Graphics on Nov. 9 and will be presented at SIGGRAPH Asia, a computer graphics conference in Kuala Lumpur, Malaysia, from Dec. 1 to 4.
