EMO: Emote Portrait Alive – Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions [Ref] [Ref]
Vocal Source: Dua Lipa – Don’t Start Now
The landscape of image generation has undergone a seismic shift thanks to the advent of Diffusion Models, a breakthrough that’s redefining the boundaries of digital creativity. These models, distinguished for their prowess in crafting high-fidelity images, are now steering the realm of video generation into uncharted territories, particularly in human-centric videos like talking head sequences.
The Rise of Diffusion Models in Image and Video Synthesis
Diffusion Models have set new standards in image generation by leveraging large-scale datasets and a unique progressive generation technique. This method produces images with an unprecedented level of detail and realism, establishing new benchmarks in the generative model domain. The potential of these models extends beyond static images, making significant strides in video generation. This exploration is not just about creating any video content but aims to generate dynamic and engaging visual narratives, showcasing the extensive capabilities of Diffusion Models.
Human-Centric Video Generation: The Challenge of Talking Head Videos
Creating talking head videos, where facial expressions are generated from audio clips, presents intricate challenges. These videos require the generation of nuanced and diverse human facial movements, a task traditional methods have attempted to simplify by imposing constraints, such as using 3D models for facial keypoints or guiding motions with pre-existing video sequences. While these techniques reduce complexity, they often compromise the naturalness and richness of the facial expressions produced.
Introducing a New Framework for Expressive Talking Head Videos
Their ambition is to develop a talking head framework that not only captures a wide range of realistic facial expressions, including subtle micro-expressions, but also supports natural head movements. By employing Diffusion Models, our method directly synthesizes character head videos from images and audio clips without needing intermediate representations. This streamlined approach facilitates the creation of talking head videos with high visual and emotional fidelity, closely mirroring the expressive nuances present in the audio input.
Overcoming the Challenges of Audio-Visual Integration
Integrating audio with Diffusion Models to generate expressive facial movements is complex, primarily due to the ambiguous mapping between audio cues and facial expressions. This ambiguity can lead to instability, such as facial distortions or jittering between frames. To tackle this, we’ve implemented stable control mechanisms within our model, including a speed controller and a face region controller. These controls act as hyperparameters, ensuring the final videos maintain diversity and expressiveness without compromising stability.
Ensuring Consistency with the Input Image
To guarantee the character’s identity remains consistent throughout the video, we’ve adopted and improved upon the ReferenceNet approach. Our FrameEncoding module is designed to preserve the character’s identity across frames, ensuring a seamless and coherent video output that faithfully represents the input reference image.
A Robust Training Dataset for Enhanced Model Performance
To train our model effectively, we’ve compiled an extensive dataset featuring over 250 hours of footage and more than 150 million images. This dataset spans a variety of content, including speeches, film and television clips, and singing performances in multiple languages. The diversity of our dataset ensures our model can capture a wide range of human expressions and vocal styles, laying a solid foundation for our EMO framework.
Setting New Benchmarks in Video Generation
Their model has been rigorously tested against the HDTF dataset, where it outperformed current state-of-the-art methods like DreamTalk, Wav2Lip, and SadTalker across various metrics. Through extensive experiments, user studies, and qualitative evaluations, our approach has proven capable of generating highly natural and expressive talking and even singing videos, achieving unparalleled results.
Conclusion
The introduction of Diffusion Models into the video generation sphere marks a significant milestone, particularly for human-centric video synthesis. Our innovative framework for generating talking head videos exemplifies how leveraging these models can lead to the production of highly expressive and realistic videos. As we continue to refine and enhance this technology, the possibilities for creative and engaging video content are boundless, heralding a new era in digital media creation.