This article is also available in:
French
French company EISKO, specialized in digital doubles and 3D humans, has unveils a deepfake video showing Sean Connery replaced by Jean Dujardin in Dr No.
We met with EISKO in their offices to know more about this project, their face replacement technology, as well as their plans for the future.
Here is the video, followed by more information.
Table of contents
How to create a deepfake: AI and compositing hand in hand
The deepfake workflow used by EISKO relies on three main steps:
- Reference pictures of the subject are gathered. Videos, most of the time. Individual pictures are then extracted and ingested by the AI to train it. The goal is to provide the AI pictures in various lighting conditions, with varying facial expressions. Of course, you also have to avoid references covering a very wide span of time: using sequences from the first movies starring Jean Dujardin would be a bad idea if you want your deepfake to look like he currently does.
- The AI is then fed the video sequence to be used as an input. The AI will then infer (i.e. create) the deepfake.
- Last, but not least, compositing is needed to add film grain, adjust the colors and shadows.
According to EISKO, their AI is quite robust: it won’t create deepfakes that suddenly look nothing like the person they are supposed to fake. Moreover, shading and colors are already quite good out of the box. Compositing will, however, remain a key step of the process for the time being. Touch-ups are sometimes required beyond color adjustments: for example, the gaze sometimes needs to be tweaked a little bit.
It should be noted that even though the tech demo does not include spoken dialogue, this is just a creative choice, not a limitation: deepfakes created by EISKO can handle dialogue.
In order to create this proprietary technology, EISKO CEO Cedric Guiard set up a team dedicated to AI within EISKO. Three employees are at the heart of this endeavour: Cesar Herrmann, Romain Reinert, and EISKO CTO Louis Rognon. The first two are engineers speciazized in machine learning and AI, while the third has a background in optics and computer vision.
Limitations and upcoming challenges
Of course, even though the tech demo showcased by EISKO is already impressive, there is still room for improvement.
Let’s focus on performance. As usual with this kind of AI, training is the most computer-hungry step. EISKO explained us that they currently need about a week of training to be able to create deepfakes of someone, sometimes 2 or 3 depending on the resolution they want to achieve (up to 1024 pixels in width/heigth so far). Depending on whether you need close-up shots or not, for example, you can therefore save some time. The R&D team is of course working on speeding up the training process.
Once the AI is trained, though, producting the deepfake itself (the inference step) is way faster, around 100ms per frame. this is good enough for artists to work in near real-time conditions, but still not fast enough for live broadcast.
Of course, upcoming GPUs will make both step even faster.
We should also highlight that EISKO can only do face replacement, not head replacement: the hair from the video used as an input remains unchanged. Using deepfakes to create hair could be done, but handling the border between the hair and the background is way harder. EISKO explains that this limitation isn’t really an issue for the Media & Entertainement market. When working on French TV show Hôtel du Temps for example, which features deepfakes of famous celebrities, VFX studio Mac Guff and the production team decided to use actors and actresses with wigs or hairstyles that would match those of the celebrities (for more information, you can check out our article about the show).
EISKO’s secret: their training data
The secret to create a good deepfake is the quality of the training data. Which is good news for EISKO, since they have been 3D scanning humans for years using photos. They already have the hardware and know-how needed, and they intend to use it. The plan is to create a capture workflow: an actor/actress would come to their office, EISKO would shoot them under various lighting conditions and with several facial expressions, then these pictures would be used as training data.
Face replacement: old and new markets
Thanks to their proprietary technology, EISKO can provide face replacement services. But this kind of tool can also be used in other ways. For example, you can de-age an actor by training the AI on their older movies, then creating a deepfake of themselves.
EISKO won’t stop there. The company also intends to target other markets, such as the cosmetic industry. This area is already quite fond of 3D scans/digital doubles, and deepfakes can be quite useful for them as well.
EISKO also plans to use deepfakes for… Chatbots! Many chatbots use 3D avatars, which aren’t very realistic. The idea would be to apply deepfakes onto these 3D avatars, in order to create photorealistic characters. And even if we mentionned earlier that EISKO’s deepfakes aren’t ready for live broadcast yet, you can tweak things behind the scenes, play with resolution and fidelity in order to get better performance.
Here is another tech demo published by EISKO on LinkedIn: they turned their own team into Jean Dujardin.
We also asked EISKO whether they were considering providing this technology to other studios. They are thinking about it, but nothing is decided at this point.
Ethics and legal support: the keys to success?
AI and deepfake cause a lot of controversy, and righfully so. The training data and the way these tools are used raise real questions. For example, several widely used generative AI tools have been trained on pictures scrapped online without asking the creators. Furthermore, deepfakes and identity theft are a match made in heaven.
EISKO is well aware of this. CEO Cedric Guiard is therefore adamant that legal agreements are a key part of this new market. This is not surprising: EISKO has also been highlighting legal agreements for years when it comes to 3 scans of celebrities (they created digital doubles for Margot Robbie, Kylian Mbappé and Eva Green, for example). It should be highlighted, by the way, that they also include protection against deepfakes in their 3D scanning contracts if required.
EISKO explains that when it comes to face replacement, they will work on legal agreements step by step, on a per-project basis.
As for the pictures used to create these deepfakes, for upcoming commercial projects EISKO will either acquire the rights of existing movies/pictures for this specific use, or create the data thanks to their own hardware (as explained above).
After meeting with EISKO, here’s our takeaway:
- First of all, the goal is not to replace 3D scan with AI, but to create another tool alongside the existing one. And depending on the specific use case, technical expertise from 3D scanning technology can be used to feed the AI.
- Furthermore, we can expect performance improvements in the near future, thanks to software improvements as well as more powerful GPUs.
- Last, but not least, legal agreements might be a key factor for studio in order to avoid ethical issues, and to make a name for themselves in this market.
EISKO told us that more tech demos are on the way. In the meantime, here is their latest showreel, focused on digital doubles and 3D humans.
