Zeroscope V2 XL Just Dropped

A watermark-free Modelscope-based video model capable of generating high quality video at 1024 x 576. Here’s how to use it:

Imports

Import pytorch and the diffusion pipeline library, along with the export_to_video utility, from the huggingface diffusers library.

import torch
from diffusers import DiffusionPipeline
from diffusers.utils import export_to_video

Loading Pipeline

Loading the zeroscope v2 model using huggingface.

(NOTE: to get 1024x576 output replace the first line with the commented line; you will need 15.3gb of VRAM for 24 frames of output at that resolution)

pipe = DiffusionPipeline.from_pretrained("cerspense/zeroscope_v2_576w", torch_dtype=torch.float16, variant="fp16")
# pipe = DiffusionPipeline.from_pretrained("cerspense/zeroscope_v2_XL", torch_dtype=torch.float16, variant="fp16")

pipe = pipe.to("cuda")

Output Information

Here, you can enter a prompt, output path, and length of output video of your choice.

prompt = "Car driving in city at night"
output_path = "/content/video.mp4"
num_frames = 24

Inference

The video is generated in this cell and exported as an mp4 file.

(NOTE: If you are generating 1024x576 output, replace the first line with the commented line again)

video_frames = pipe(prompt=prompt, num_frames=num_frames, height=320, width=576).frames
# video_frames = pipe(prompt=prompt, num_frames=num_frames, height=576, width=1024).frames

video_path = export_to_video(video_frames, output_path)
video_path

'/content/video.mp4'

Playing the Video

A simple function to display the video in the notebook.

import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

def display_video(video):
    fig = plt.figure(figsize=(3,3))

    mov = []
    for i in range(len(video)):
        img = plt.imshow(video[i], animated=True)
        plt.axis('off')
        mov.append([img])

    anime = animation.ArtistAnimation(fig, mov, interval=100, repeat_delay=1000)

    plt.close()
    return anime

HTML(display_video(video_frames).to_html5_video())

And you’re done! For more information, check out the following links:

576x320 model: https://huggingface.co/cerspense/zeroscope_v2_576w

1024x576: https://huggingface.co/cerspense/zeroscope_v2_XL