文字前處理入門

Deep Learning for Text with PyTorch

Shubham Jain

Data Scientist

你將學到什麼

  • 文字分類
  • 文字生成
  • 編碼
  • 文字的深度學習模型
  • Transformer 架構
  • 模型防護

應用情境：

  • 情感分析
  • 摘要生成
  • 機器翻譯

Sentiment Analysis

Deep Learning for Text with PyTorch

你需要先懂什麼

先修課程：Intermediate Deep Learning with PyTorch

  • 使用 PyTorch 的深度學習模型
  • 訓練與評估迴圈
  • 卷積神經網路（CNN）與遞迴神經網路（RNN）
Deep Learning for Text with PyTorch

文字處理流程

 

 

Pytorch Processing Pipeline

Deep Learning for Text with PyTorch

文字處理流程

 

 

Pytorch Processing Pipeline

 

  • 清理並準備文字
Deep Learning for Text with PyTorch

PyTorch 與 NLTK

PyTorch Logo

NLTK Logo

  • Natural Language Toolkit
    • 將原始文字轉為處理後文字
Deep Learning for Text with PyTorch

前處理技巧

  • 斷詞（Tokenization）
  • 停用詞移除
  • 詞幹提取（Stemming）
  • 罕見詞移除
Deep Learning for Text with PyTorch

斷詞（Tokenization）

  • 從文字中擷取 token 或單字
  • 使用 torchtext 進行斷詞
from torchtext.data.utils import get_tokenizer

tokenizer = get_tokenizer("basic_english")

tokens = tokenizer("I am reading a book now. I love to read books!")
print(tokens)

["I", "am", "reading", "a", "book", "now", ".", "I", "love", "to", "read", 
"books", "!"]
Deep Learning for Text with PyTorch

停用詞移除

  • 移除對語意貢獻不大的常見詞
  • 停用詞："a"、"the"、"and"、"or" 等
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

tokens = ["I", "am", "reading", "a", "book", "now", ".", "I", "love", "to", "read",
"books", "!"]
filtered_tokens = [token for token in tokens if token.lower() not in stop_words]

print(filtered_tokens)

["reading", "book", ".", "love", "read", "books", "!"]
Deep Learning for Text with PyTorch

詞幹提取（Stemming）

  • 將單字簡化為詞幹或基本形式
  • 例如：「running」、「runs」、「ran」變為 run
import nltk
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()

filtered_tokens = ["reading", "book", ".", "love", "read", "books", "!"]

stemmed_tokens = [stemmer.stem(token) for token in filtered_tokens]

print(stemmed_tokens)

["read", "book", ".", "love", "read", "book", "!"]
Deep Learning for Text with PyTorch

罕見詞移除

  • 移除少見且無助於模型的單字
from nltk.probability import FreqDist
stemmed_tokens= ["read", "book", ".", "love", "read", "book", "!"]  
freq_dist = FreqDist(stemmed_tokens)

threshold = 2

common_tokens = [token for token in stemmed_tokens if freq_dist[token] > threshold]
print(common_tokens)

["read", "book", "read", "book"]
Deep Learning for Text with PyTorch

前處理技巧

斷詞、停用詞移除、詞幹提取與罕見詞移除

  • 降低特徵數
  • 更乾淨且具代表性的資料集
  • 還有更多技巧可用
Deep Learning for Text with PyTorch

一起來練習吧！

Deep Learning for Text with PyTorch

Preparing Video For Download...