计算机视觉

使用 Hugging Face 的多模态模型

James Chapman

Curriculum Manager, DataCamp

视觉模型

视觉任务示意图

¹ https://arxiv.org/abs/1409.1556

分类

from datasets import load_dataset
dataset = load_dataset("nlphuji/flickr30k")
image = dataset['test'][134]["image"]

一场棒球比赛的图像

from transformert import pipeline
pipe = pipeline("image-classification", 
"google/mobilenet_v2_1.0_224") # 224x224 input


pred = pipe(image)
print("Predicted class:", pred[0]['label'])

Predicted class: ballplayer, baseball player

目标检测

dataset['test'][52]["image"]

武术比赛，尚无边界框

目标检测

pipe = pipeline("object-detection", "facebook/detr-resnet-50", revision="no_timm")

outputs = pipe(image, threshold=0.95)


for obj in outputs:
    box = obj['box']

    print(f"Detected {obj['label']} with confidence {obj['score']:.2f}
    at ({box['xmin']}, {box['ymin']}) to ({box['xmax']}, {box['ymax']})")

Detected person with confidence 0.97 at (381, 131) to (499, 330)
Detected person with confidence 0.96 at (381, 36) to (427, 103)
Detected person with confidence 0.98 at (253, 39) to (294, 125)
Detected person with confidence 1.00 at (144, 36) to (296, 170)
Detected person with confidence 0.95 at (280, 60) to (399, 294)

目标检测

import matplotlib.pyplot as plt
import matplotlib.patches as patches
ax = plt.gca()
colors = ['r', 'g', 'b', 'y', 'm', 'c', 'k']

plt.imshow(image)

for n, obj in enumerate(outputs):
    box = obj['box']

    rect = patches.Rectangle(
      (box['xmin'], box['ymin']), 
      box['xmax']-box['xmin'], 
      box['ymax']-box['ymin'],

      linewidth=1, 
      edgecolor=colors[n], facecolor='none')

    ax.add_patch(rect)

plt.show()

武术比赛，带有边界框

分割

语义分割

输出：与输入同尺寸的二维数组
去背景：每个像素为1（前景）或0（背景）
图像 × 输出 → 去背景的图像

分割

pipe = pipeline("image-segmentation", 
                model="briaai/RMBG-1.4", 
                trust_remote_code=True)

outputs = pipe(image)


plt.imshow(outputs)
plt.show()

武术比赛图像，背景已移除

让我们来练习！

使用 Hugging Face 的多模态模型