Menyiapkan data untuk RLHF

Reinforcement Learning from Human Feedback (RLHF)

Mina Parham

AI Engineer

Himpunan data Preferensi vs. Prompt

Proses RLHF lengkap.

Reinforcement Learning from Human Feedback (RLHF)

Himpunan data Preferensi vs. Prompt

Proses RLHF lengkap dengan himpunan data prompt awal.

Reinforcement Learning from Human Feedback (RLHF)

Himpunan data Preferensi vs. Prompt

Proses RLHF lengkap dengan himpunan data prompt awal dan himpunan data preferensi yang masuk ke model reward.

Reinforcement Learning from Human Feedback (RLHF)

Himpunan data prompt

  • Pertanyaan untuk model
  • Tersedia di Hugging Face datasets
prompt_data = load_dataset("center-for-humans-and-machines/rlhf-hackathon-prompts", 
                           split="train")
prompt_data['prompt'][0]

'How important is climate change?'
  • Mungkin perlu mengekstrak prompt
  • Cari penanda seperti: Input=, {{Text}}:, ###Human:
Reinforcement Learning from Human Feedback (RLHF)

Menjelajahi himpunan data preferensi

from datasets import load_dataset
preference_data = load_dataset("qgallouedec/hh-rlhf-helpful-base-trl-style", 
                               split="train")
Reinforcement Learning from Human Feedback (RLHF)

Memproses himpunan data preferensi

def extract_prompt(text):
    # Extract the prompt as the first element in the list
    prompt = text[0]["content"]
    return prompt

# Apply the extraction function to the dataset
preference_data_with_prompt = preference_data.map(
    lambda example: {**example, 'prompt': extract_prompt(example['chosen'])}
)
  • Cara mengekstrak prompt berbeda untuk tiap himpunan data
Reinforcement Learning from Human Feedback (RLHF)

Himpunan data preferensi final

sample = preference_data_with_prompt.select(range(1))
sample['prompt']

'What vitamins are essential for the body to function?'

sample['chosen']

[ { "content": "What vitamins are essential for the body to function?", "role": 
   "user" }, { "content": "There are some very important vitamins that ensure the 
   proper functioning of the body, including Vitamins A, C, D, E, and K along ...}]
Reinforcement Learning from Human Feedback (RLHF)

Ayo berlatih!

Reinforcement Learning from Human Feedback (RLHF)

Preparing Video For Download...