為 RLHF 準備資料

Reinforcement Learning from Human Feedback（RLHF）

Mina Parham

AI Engineer

偏好資料集 vs. 提示資料集

完整的 RLHF 流程。

Reinforcement Learning from Human Feedback（RLHF）

偏好資料集 vs. 提示資料集

含初始提示資料集的完整 RLHF 流程。

Reinforcement Learning from Human Feedback（RLHF）

偏好資料集 vs. 提示資料集

含初始提示資料集與導入獎勵模型的偏好資料集之完整 RLHF 流程。

Reinforcement Learning from Human Feedback（RLHF）

提示資料集

  • 給模型的問題
  • 可在 Hugging Face 資料集找到
prompt_data = load_dataset("center-for-humans-and-machines/rlhf-hackathon-prompts", 
                           split="train")
prompt_data['prompt'][0]

'How important is climate change?'
  • 可能需要從欄位中擷取提示
  • 尋找標記，例如：Input=、{{Text}}:、###Human:
Reinforcement Learning from Human Feedback（RLHF）

探索偏好資料集

from datasets import load_dataset
preference_data = load_dataset("trl-internal-testing/hh-rlhf-helpful-base-trl-style", 
                               split="train")

Hugging Face 平台中 hh-rlhf 資料集的截圖。

Reinforcement Learning from Human Feedback（RLHF）

處理偏好資料集

def extract_prompt(text):
    # Extract the prompt as the first element in the list
    prompt = text[0]["content"]
    return prompt

# Apply the extraction function to the dataset
preference_data_with_prompt = preference_data.map(
    lambda example: {**example, 'prompt': extract_prompt(example['chosen'])}
)
  • 不同資料集的提示擷取方式會不一樣
Reinforcement Learning from Human Feedback（RLHF）

最終的偏好資料集

sample = preference_data_with_prompt.select(range(1))
sample['prompt']

'What vitamins are essential for the body to function?'

sample['chosen']

[ { "content": "What vitamins are essential for the body to function?", "role": 
   "user" }, { "content": "There are some very important vitamins that ensure the 
   proper functioning of the body, including Vitamins A, C, D, E, and K along ...}]
Reinforcement Learning from Human Feedback（RLHF）

一起來練習吧！

Reinforcement Learning from Human Feedback（RLHF）

Preparing Video For Download...