Real-Time Face Capture Software Will Make Donald Trump Your Digital Puppet
New program can manipulate mouths and eyebrows but struggles with hands.
Millennials! They’re now hijacking the YouTube videos of Vladimir Putin and Donald Trump to force silly mouth motions upon them. And it is a delight.
An updated program from German and Californian researchers can manipulate YouTube videos in real time, translocating fishy gasps from graduate students onto George Bush’s face.
It builds on earlier work that Stanford University visiting computer scientist Matthias Niessner describes to the New York Times as “live facial re-enactment.”
In short, the algorithm trains itself on a segment of target video (say, a clip of Arnold Schwarzenegger talking) and creates a corresponding map of pixels for the source face (like a grad student) from a web cam. The identity, illumination, and pose of the target actor remain the same, but the algorithm steals the source’s expression and pastes it over the target.
The new program is better at rendering the inside of the target’s mouth, which it does with gnarly realism:
The scientists say their reenactment, to be presented at the computer vision conference CVPR in June, could improve movie dubbing or Skype conversations. (Coupling an automatic translation software with this image mapping technique, the idea goes, could make it seem as if two foreigners were speaking naturally in each others’ tongues.)
But just as trolls can abuse Photoshop, the potential for malicious applications isn’t lost on YouTube commenters — as one puts it: “This is too far. Very scary potential for misuse. Do no pursue this for the good of humanity.”
Luckily, the conversion isn’t 100 percent instantaneous; it takes 30 milliseconds, which is too fast for a human to detect but not, in theory, too quick for a computer program to filter out. Moreover, just as there are ways to tell if an image has been Photoshopped — warped lines and incongruous lighting are telltale signs — the Stanford University researchers believe similar detection techniques would arise if such live reenactment software became widespread.
In the meantime, if you’re a speaker at risk of becoming a plaything for a Gen-Y’er with a head full of computer knowhow and a heart full of student loan debt, avoid getting facially commandeered by continuously waving your hand in front of your face or jerking your head side to side. The re-enactment program can neither figure out fingers nor handle motion beyond a 30-degree angle from the neck. Here’s to more State of the Union headbanging.
