Auto 模型與 Tokenizer

善用 Hugging Face

Jacob H. Marquez

Lead Data Engineer

Pipeline：快速又簡單

from transformers import pipeline  

my_pipeline = pipeline(
    "text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english"))

print(my_pipeline("Wi-Fi is slower than a snail today!"))

[{'label': 'NEGATIVE', 'score': 0.99}]

Auto 類別：彈性又強大

Auto 類別：彈性存取模型與 tokenizer
更高控制：掌握模型行為與輸出
進階任務首選

Pipeline＝快速；Auto 類別＝彈性

三個帶切換的滑桿與一隻手在調整，象徵更高的控制度。

AutoModels

選擇 AutoModel 類別直接下載模型

from transformers import AutoModelForSequenceClassification


# Download a pre-trained text classification model
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)

AutoTokenizers

準備文字輸入資料
建議使用「與模型成對的 tokenizer」

from transformers import AutoTokenizer


# Retrieve the tokenizer paired with the model
tokenizer = AutoTokenizer.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)

用 AutoTokenizer 斷詞

Tokenizer 會清理輸入並將文字切成 token

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")


# Tokenize input text
tokens = tokenizer.tokenize("AI: Helping robots think and humans overthink:)")
print(tokens)

['ai', ':', 'helping', 'robots', 'think', 'and', 
 'humans', 'over', '##thi', '##nk', ':', ')']

模型不同，tokenizer 也不同

本課模型（distilbert-base-uncased）：

['ai', ':', 'helping', 'robots', 'think', 'and', 'humans', 'over', '##thi',
'##nk', ':', ')']

BERT-Base-Cased Tokenizer：

['AI', ':', 'Help', '##ing', 'robots', 'think', 'and', 'humans', 'over',
'##thin', '##k', ':', ')']

用 Auto 類別打造自訂 Pipeline

from transformers import AutoModelForSequenceClassification,
AutoTokenizer, pipeline


# Download the model and tokenizer
my_model = AutoModelForSequenceClassification.from_pretrained(
  "distilbert-base-uncased-finetuned-sst-2-english")
my_tokenizer = AutoTokenizer.from_pretrained(
  "distilbert-base-uncased-finetuned-sst-2-english")


# Create the custom pipeline
my_pipeline = pipeline(
  task="sentiment-analysis", model=my_model, tokenizer=my_tokenizer)

AutoModels 與 AutoTokenizers 的應用情境

🔧 用於更高控制與客製化
📝 文字前處理： 依情境清理與斷詞
🏆 閾值設定： 分類時凸顯關鍵類別
🚀 複雜流程： 控制多階段處理與整合

$$ 更多控制與客製化

一起來練習吧！

善用 Hugging Face