電腦視覺

使用 Hugging Face 的多模態模型

James Chapman

Curriculum Manager, DataCamp

視覺模型

視覺任務示意圖

¹ https://arxiv.org/abs/1409.1556

分類

from datasets import load_dataset
dataset = load_dataset("nlphuji/flickr30k")
image = dataset['test'][134]["image"]

棒球比賽影像

from transformert import pipeline
pipe = pipeline("image-classification", 
"google/mobilenet_v2_1.0_224") # 224x224 input


pred = pipe(image)
print("Predicted class:", pred[0]['label'])

Predicted class: ballplayer, baseball player

物件偵測

dataset['test'][52]["image"]

武術比賽（含邊界框）

物件偵測

pipe = pipeline("object-detection", "facebook/detr-resnet-50", revision="no_timm")

outputs = pipe(image, threshold=0.95)


for obj in outputs:
    box = obj['box']

    print(f"Detected {obj['label']} with confidence {obj['score']:.2f}
    at ({box['xmin']}, {box['ymin']}) to ({box['xmax']}, {box['ymax']})")

Detected person with confidence 0.97 at (381, 131) to (499, 330)
Detected person with confidence 0.96 at (381, 36) to (427, 103)
Detected person with confidence 0.98 at (253, 39) to (294, 125)
Detected person with confidence 1.00 at (144, 36) to (296, 170)
Detected person with confidence 0.95 at (280, 60) to (399, 294)

物件偵測

import matplotlib.pyplot as plt
import matplotlib.patches as patches
ax = plt.gca()
colors = ['r', 'g', 'b', 'y', 'm', 'c', 'k']

plt.imshow(image)

for n, obj in enumerate(outputs):
    box = obj['box']

    rect = patches.Rectangle(
      (box['xmin'], box['ymin']), 
      box['xmax']-box['xmin'], 
      box['ymax']-box['ymin'],

      linewidth=1, 
      edgecolor=colors[n], facecolor='none')

    ax.add_patch(rect)

plt.show()

武術比賽（顯示邊界框）

分割

語意分割

輸出：與輸入相同尺寸的 2D 陣列
去背：每個像素為 1（前景）或 0（背景）
影像 $\times$ 輸出 → 去背後的影像

分割

pipe = pipeline("image-segmentation", 
                model="briaai/RMBG-1.4", 
                trust_remote_code=True)

outputs = pipe(image)


plt.imshow(outputs)
plt.show()

武術比賽影像（已去除背景）

一起來練習吧！

使用 Hugging Face 的多模態模型