文字分類的評估指標

Deep Learning for Text with PyTorch

Shubham Jain

Instructor

為何評估指標重要

聚焦書評：

  • 想像一個模型用來判斷書評的情感
  • 模型聲稱暢銷書評價很差。要接受嗎？
  • 使用評估指標

書評

Deep Learning for Text with PyTorch

評估 RNN 模型

# Initialize model, criterion, and optimizer
rnn_model = RNNModel(input_size, hidden_size, num_layers, num_classes)
...
# Model training
for epoch in range(10): 
    outputs = rnn_model(X_train)
    ...
    print(f'Epoch: {epoch+1}, Loss: {loss.item()}')


outputs = rnn_model(X_test) 
_, predicted = torch.max(outputs, 1)
Deep Learning for Text with PyTorch

Accuracy（準確率）

  • 正確預測數占所有預測的比例
from torchmetrics import Accuracy

actual = torch.tensor([0, 1, 1, 0, 1, 0])
predicted = torch.tensor([0, 0, 1, 0, 1, 1])

accuracy = Accuracy(task="binary", num_classes=2)

acc = accuracy(predicted, actual)
print(f"Accuracy: {acc}")

Accuracy: 0.6666666666666666
Deep Learning for Text with PyTorch

不只看準確率

  • 10,000 則書評：9,800 則為正向
    • 一個永遠預測正向的模型：98% 準確率
      • 但模型無法辨識負向書評

 

  • Precision：把書評判為負向時的可靠度
  • Recall：模型找出負向書評的能力
  • F1 Score：Precision 與 Recall 的折衷
Deep Learning for Text with PyTorch

Precision 與 Recall

  • Precision：正確預測為正類的數量／被預測為正類的總數
  • Recall：正確預測為正類的數量／實際正類的總數
from torchmetrics import Precision, Recall

precision = Precision(task="binary", num_classes=2)
recall = Recall(task="binary", num_classes=2)

prec = precision(predicted, actual)
rec = recall(predicted, actual)

print(f"Precision: {prec}")
print(f"Recall: {rec}")

Precision: 0.6666666666666666
Recall: 0.5
Deep Learning for Text with PyTorch

Precision 與 Recall

Precision: 0.6666666666666666
Recall: 0.5
  • Precision：有 66.66% 正確判為正類
  • Recall：捕捉到 50% 的正類
Deep Learning for Text with PyTorch

F1 分數

  • 協調 Precision 與 Recall
  • 適合類別不平衡時評估
from torchmetrics import F1Score
f1 = F1Score(task="binary", num_classes=2)
f1_score = f1(predicted, actual)
print(f"F1 Score: {f1_score}")

F1 Score: 0.5714285714285715
  • F1 分數為 1＝Precision 與 Recall 皆完美
  • F1 分數為 0＝表現最差
Deep Learning for Text with PyTorch

重點考量

  • 多類別分數可能相同

    • 可能代表模型表現良好

  • 解讀結果時務必結合問題情境！

Deep Learning for Text with PyTorch

一起來練習吧！

Deep Learning for Text with PyTorch

Preparing Video For Download...