Video oluşturma

Hugging Face ile Multi-Modal Modeller

James Chapman

Curriculum Manager, DataCamp

Video oluşturma

¹ https://link.springer.com/article/10.1007/s11263-024-02271-9

Video oluşturma

import torch
from diffusers import CogVideoXPipeline


pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-2b",
    torch_dtype=torch.float16
)


pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()

pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

CogVideoX modelinden örnek video: bir astronotun uzaylıyla el sıkışması istemiyle

¹ https://huggingface.co/THUDM/CogVideoX-2b

Video oluşturma

prompt = "A majestic lion in a sunlit African savanna, sitting regally 
on a rock formation. Golden sunlight illuminates its magnificent mane, 
then a big smile appears on its face"

video = pipe(
    prompt=prompt,

    num_inference_steps=20,

    num_frames=20,

    guidance_scale=6,

    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]

Video oluşturma

from diffusers.utils import export_to_video
from moviepy.editor import VideoFileClip


video_path = export_to_video(video, 
                             "output.mp4", 
                             fps=8)
video = VideoFileClip(video_path)

video.write_gif("video.gif")

Önceki istemden gülümseyen aslanın GIF'i

Nicel analiz

Videolarda isteme uyum zordur
CLIP olası bir strateji sunar:

CLIP video puanı karşılaştırma yaklaşımı diyagramı

Nicel analiz

from diffusers.utils import load_video
from torchmetrics.functional.multimodal import clip_score
from functools import partial

frames = load_video(video_path)
clip_score_fn = partial(clip_score, model_name_or_path="openai/clip-vit-base-patch16")

scores = []
for frame in frames:
  frame_int = np.array(frame).astype("uint8")
  frame_tensor = torch.from_numpy(frame_int).unsqueeze(0).permute(0, 3, 1, 2)

  score = clip_score_fn(frame_tensor, [prompt]).detach()
  scores.append(float(score))
avg_clip_score = round(np.mean(scores), 4)
print(f"Average CLIP score: {avg_clip_score}")

Average CLIP score: 30.6274

Ayo berlatih!

Hugging Face ile Multi-Modal Modeller