用於文字生成的預訓練模型

Deep Learning for Text with PyTorch

Shubham Jain

Instructor

為何選擇預訓練模型？

優點

  1. 以大型資料集訓練
  2. 多種文字生成任務表現佳
    • 情感分析
    • 文字補全
    • 語言翻譯

侷限

  1. 訓練計算成本高
  2. 儲存空間需求大
  3. 自訂彈性有限
Deep Learning for Text with PyTorch

PyTorch 中的預訓練模型

  • Hugging Face Transformers：預訓練模型函式庫
  • 預訓練模型：
    • GPT-2
    • T5

HuggingFace 標誌

Deep Learning for Text with PyTorch

認識 GPT-2 的 Tokenizer 與模型

GPT2LMHeadModel：

  • HuggingFace 的 GPT-2 實作
  • 針對文字生成設計

GPT2Tokenizer：

  • 將文字轉為 token
  • 支援次詞切分：'larger' 可能變成 ['large', 'r']
Deep Learning for Text with PyTorch

GPT-2：文字生成實作

import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

seed_text = "Once upon a time"

input_ids = tokenizer.encode(seed_text, return_tensors='pt')
Deep Learning for Text with PyTorch

GPT-2：文字生成實作 II

output = model.generate(

                                                           )
Deep Learning for Text with PyTorch

GPT-2：文字生成實作 II

output = model.generate(input_ids, max_length=40, 

                                                           )
Deep Learning for Text with PyTorch

GPT-2：文字生成實作 II

output = model.generate(input_ids, max_length=40, temperature=0.7,     

                                                           )
Deep Learning for Text with PyTorch

GPT-2：文字生成實作 II

output = model.generate(input_ids, max_length=40, temperature=0.7,     
                        no_repeat_ngram_size=2, 
                                                           )
Deep Learning for Text with PyTorch

GPT-2：文字生成實作 II

output = model.generate(input_ids, max_length=40, temperature=0.7,     
                        no_repeat_ngram_size=2, 
                        pad_token_id=tokenizer.eos_token_id)
Deep Learning for Text with PyTorch

GPT-2：文字生成輸出

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Generated Text: Once upon a time, the world was a place of great beauty
and great danger. The world of the gods was the place where the great gods were 
born, and where they were to live.
Deep Learning for Text with PyTorch

T5：語言翻譯實作

  • t5-small：Text-to-Text Transfer Transformer
  • 預訓練模型，適用於語言翻譯任務
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_prompt = "translate English to French: 'Hello, how are you?'"

input_ids = tokenizer.encode(input_prompt, return_tensors="pt")

output = model.generate(input_ids, max_length=100)
Deep Learning for Text with PyTorch

T5：語言翻譯輸出

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)

print("Generated text:",generated_text)

Generated text:
"Bonjour, comment êtes-vous?"
Deep Learning for Text with PyTorch

如何選擇合適的預訓練模型

  • 選擇很多！

 

  • GPT-2：文字生成
  • DistilGPT-2（GPT-2 的精簡版）：文字生成
  • BERT：文字分類、問答
  • T5（t5-small 為 T5 的精簡版）：語言翻譯、摘要

 

  • 可在 HuggingFace 與其他資源庫找到
Deep Learning for Text with PyTorch

一起來練習吧！

Deep Learning for Text with PyTorch

Preparing Video For Download...