모델 성능 평가
Keras로 배우는 언어 모델링을 위한 순환 신경망(RNN)
David Cecchini
Data Scientist
정확도만으로는 부족합니다
20개 클래스에서 정확도 80%. 모델이 좋을까요?
- 모든 클래스를 제대로 분류하나요?
- 클래스별 정확도가 같은가요?
- 다수 클래스에 과적합되었나요?
잘 모르겠습니다!
혼동 행렬
클래스별 실제값과 예측값 확인
정밀도
정밀도(Precision)
$$\text{Precision}_{\text{class}} = \frac{\text{Correct}_{\text{class}}}{\text{Predicted}_{\text{class}}}$$
예시:
$$ \text{Precision}_{\text{sci.space}} = \frac{76}{76+7+9} = 0.83 $$ $$ \text{Precision}_{\text{alt.atheism}} = \frac{1}{2+1+0} = 0.33 $$ $$ \text{Precision}_{\text{soc.religion.christian}} = \frac{3}{0+2+3} = 0.60 $$
재현율
재현율(Recall)
$$\text{Recall}_{\text{class}} = \frac{\text{Correct}_{class}}{N_\text{class}}$$
예시:
$$ \text{Recall}_{\text{sci.space}} = \frac{76}{76+2+0} = 0.97 $$ $$ \text{Recall}_{\text{alt.atheism}} = \frac{1}{7+1+2} = 0.10 $$ $$ \text{Recall}_{\text{soc.religion.christian}} = \frac{3}{9+0+3} = 0.25 $$
F1-점수
F1-점수
$$\text{F1 score} = 2 * \frac{\text{precision}_{\text{class}} * \text{recall}_{\text{class}}}{\text{precision}_{\text{class}} + \text{recall}_{\text{class}}}$$
예시:
$$ f1score_{sci.space} = 2 \frac{0.83 * 0.97}{0.83 + 0.97} = 0.89 $$ $$ f1score_{alt.atheism} = 2 \frac{033 * 0.10}{033 + 0.10} = 0.15 $$ $$ f1score_{soc.religion.christian} = 2 \frac{060 * 0.25}{060 + 0.25} = 0.35 $$
Sklearn 혼동 행렬
from sklearn.metrics import confusion_matrix# Build the confusion matrix confusion_matrix(y_true, y_pred)
Output:
array([[76, 2, 0],
[ 7, 1, 2],
[ 9, 0, 3]], dtype=int64)
성능 지표
sklearn의 지표
# Functions of sklearn
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
성능 지표
# Accuracy
print(accuracy_score(y_true, y_pred))
$ 0.80
precision, recall, f1 함수에 average=None을 추가하세요
print(precision_score(y_true, y_pred, average=None))
print(recall_score(y_true, y_pred, average=None))
print(f1_score(y_true, y_pred, average=None))
$ array([0.83, 0.33, 0.60])
$ array([0.97, 0.10, 0.25])
$ array([0.89, 0.15, 0.35])
분류 보고서
한 번에 요약:
lab_names = ['sci.space', 'alt.atheism', 'soc.religion.christian']
print(classification_report(y_true, y_pred, target_names=lab_names))
precision recall f1-score support
sci.space 0.83 0.97 0.89 78
alt.atheism 0.33 0.10 0.15 10
soc.religion.christian 0.60 0.25 0.35 12
micro avg 0.80 0.80 0.80 100
macro avg 0.59 0.44 0.47 100
weighted avg 0.75 0.80 0.76 100
Ayo berlatih!
Keras로 배우는 언어 모델링을 위한 순환 신경망(RNN)
