The research paper “FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis” focuses on solving the challenges of video-to-video (V2V) synthesis, particularly the problem of maintaining temporal consistency across video frames. This issue is important in the context of applying image-to-image (I2I) compositing models to videos that frequently experience pixel flickering between frames.
The solution proposed in the paper is a new V2V synthesis framework called FlowVid. Developed by researchers at the University of Texas at Austin and Meta GenAI, FlowVid uniquely combines spatial conditions in the source video with temporal optical flow cues. This approach allows us to generate temporally consistent videos from the input video and text prompts. This model works seamlessly with the existing I2I model, demonstrating flexibility and efficiency by facilitating a variety of modifications, including styling, object exchange, and local editing.
FlowVid outperforms existing models such as CoDeF, Rerender, and TokenFlow in terms of synthesis efficiency. For example, it takes only 1.5 minutes to generate a 4-second video at 512×512 resolution at 30 FPS, which is much faster than the models mentioned. FlowVid also ensures high-quality output, according to user studies, which makes it preferred over other models.
FlowVid’s framework includes training using joint spatiotemporal conditions using an edit-propagate procedure for generation. This model allows you to edit the first frame using the popular I2I model and then propagate these edits to successive frames to maintain consistency and quality.
Researchers conducted extensive experiments and evaluations to demonstrate FlowVid’s effectiveness. This included qualitative and quantitative comparisons with state-of-the-art methods, user studies, and analysis of the runtime efficiency of the model. Results consistently showed that FlowVid provides a robust and efficient approach to V2V compositing, solving the long-standing challenge of maintaining temporal consistency of video frames.
For more information and a comprehensive understanding of the methodology and results, you can access the full paper at this URL: https://huggingface.co/papers/2312.17681.
The project webpage (https://jeff-liangf.github.io/projects/flowvid/) also provides additional insight.
Image source: Shutterstock