Menyiapkan data untuk RLHF

Reinforcement Learning from Human Feedback (RLHF)

Mina Parham

AI Engineer

Dataset preferensi vs. prompt

Proses RLHF lengkap.

Reinforcement Learning from Human Feedback (RLHF)

Dataset preferensi vs. prompt

Proses RLHF lengkap dengan dataset prompt awal.

Reinforcement Learning from Human Feedback (RLHF)

Dataset preferensi vs. prompt

Proses RLHF lengkap dengan dataset prompt awal dan dataset preferensi untuk melatih reward model.

Reinforcement Learning from Human Feedback (RLHF)

Dataset prompt

  • Pertanyaan untuk model
  • Tersedia di Hugging Face Datasets
prompt_data = load_dataset("center-for-humans-and-machines/rlhf-hackathon-prompts", 
                           split="train")
prompt_data['prompt'][0]

'How important is climate change?'
  • Mungkin perlu mengekstrak prompt
  • Cari penanda seperti: Input=, {{Text}}:, ###Human:
Reinforcement Learning from Human Feedback (RLHF)

Menjelajahi dataset preferensi

from datasets import load_dataset
preference_data = load_dataset("trl-internal-testing/hh-rlhf-helpful-base-trl-style", 
                               split="train")

Tangkapan layar platform Hugging Face yang menampilkan dataset hh-rlhf.

Reinforcement Learning from Human Feedback (RLHF)

Memroses dataset preferensi

def extract_prompt(text):
    # Extract the prompt as the first element in the list
    prompt = text[0]["content"]
    return prompt

# Apply the extraction function to the dataset
preference_data_with_prompt = preference_data.map(
    lambda example: {**example, 'prompt': extract_prompt(example['chosen'])}
)
  • Cara mengekstrak prompt berbeda untuk tiap dataset
Reinforcement Learning from Human Feedback (RLHF)

Dataset preferensi final

sample = preference_data_with_prompt.select(range(1))
sample['prompt']

'What vitamins are essential for the body to function?'

sample['chosen']

[ { "content": "What vitamins are essential for the body to function?", "role": 
   "user" }, { "content": "There are some very important vitamins that ensure the 
   proper functioning of the body, including Vitamins A, C, D, E, and K along ...}]
Reinforcement Learning from Human Feedback (RLHF)

Ayo berlatih!

Reinforcement Learning from Human Feedback (RLHF)

Preparing Video For Download...