Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

by Xiaoyu Shi^1*, Zhaoyang Huang^1*, Fu-Yun Wang^1*, Weikang Bian^1*, Dasong Li ¹, Yi Zhang¹, Manyuan Zhang¹, Ka Chun Cheung², Simon See², Hongwei Qin³, Jifeng Dai⁴, Hongsheng Li¹

¹CUHK-MMLab ²NVIDIA ³SenseTime ⁴ Tsinghua University

@article{shi2024motion,
            title={Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling},
            author={Shi, Xiaoyu and Huang, Zhaoyang and Wang, Fu-Yun and Bian, Weikang and Li, Dasong and Zhang, Yi and Zhang, Manyuan and Cheung, Ka Chun and See, Simon and Qin, Hongwei and others},
            journal={SIGGRAPH 2024},
            year={2024}
            }
}

Overview of Motion-I2V. The first stage of Motion-I2V targets at deducing the motions that can plausibly animate the reference image. It is conditioned on the reference image and text prompt, and predicts the motion field maps between the reference frame and all the future frames. The second stage propagates reference image’s content to synthesize frames. A novel motion-augmented temporal layer enhances 1-D temporal attention with warped features. This operation enlarges the temporal receptive field and alleviates the complexity of directly learning the complicated spatial-temporal patterns.

Usage

1. Install environments

conda env create -f environment.yaml

2. Download models

git clone https://huggingface.co/wangfuyun/Motion-I2V

3. Run the code

python -m scripts.app 

ComfyUI

ComfyUI-IG-Motion-I2V