模型評估與調整

Reinforcement Learning from Human Feedback（RLHF）

Mina Parham

AI Engineer

為何使用參考模型？

  • 毫無意義的輸出

一系列表情符號。

Reinforcement Learning from Human Feedback（RLHF）

檢查模型輸出

含回應檢查的 RLHF 流程圖。

Reinforcement Learning from Human Feedback（RLHF）

解法：KL 散度

含 KL 散度的 RLHF 流程圖。

Reinforcement Learning from Human Feedback（RLHF）

解法：KL 散度

  • 對「獎勵模型」加入「懲罰」
  • 若輸出無關，懲罰會「導回」模型
  • 「KL 散度」比較目前模型與獎勵模型

天秤圖示。

  • 介於「0」到「10」，且「不會為負」
Reinforcement Learning from Human Feedback（RLHF）

調整參數

generation_kwargs = {

    "min_length": -1, # don't ignore the EOS token

    "top_k": 0.0, # no top-k sampling

    "top_p": 1.0, 
    "do_sample": True, 
    "pad_token_id": tokenizer.eos_token_id, 
    "max_new_tokens": 32}

 

  • 參數會傳入 policy 模型
Reinforcement Learning from Human Feedback（RLHF）

檢查獎勵模型

  • 檢查獎勵模型

  • 檢查輸出（獎勵）

reward_model_results.head()

|ID | Comment                                     |Sentiment |Reward|
|---|---------------------------------------------|----------|------|
| 1 | This event was lit! So much fun!            | Positive |  0.9 |
| 2 | Terrible experience, never attending again. | Negative | -0.8 |
| 3 | It was okay, nothing extraordinary.         | Neutral  |  0.2 |
| 4 | The event was poorly organized and chaotic. | Negative | -0.85|
| 5 | Had an amazing time with great people!      | Positive |  0.95|
Reinforcement Learning from Human Feedback（RLHF）

檢查獎勵模型

  • 👍 👎 檢查極端案例
    extreme_positive = reward_model_results[reward_model_results['Reward'] >= 0.9]
extreme_negative = reward_model_results[reward_model_results['Reward'] <= -0.8]

  • 🧘 確保資料集平衡

    sentiment_distribution = reward_model_results['Sentiment'].value_counts()

  • 📊 正規化獎勵模型

    from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
scaler.fit_transform(reward_model_results[['Reward']])
Reinforcement Learning from Human Feedback（RLHF）

一起來練習吧！

Reinforcement Learning from Human Feedback（RLHF）

Preparing Video For Download...