사전 학습된 LLM 살펴보기

Reinforcement Learning from Human Feedback (RLHF)

Mina Parham

AI Engineer

파인튜닝의 중요성

RLHF 프로세스를 보여주는 다이어그램.

Reinforcement Learning from Human Feedback (RLHF)

파인튜닝의 중요성

입력과 출력을 가진 LLM을 나타내는 다이어그램.

Reinforcement Learning from Human Feedback (RLHF)

LLM 파인튜닝 단계별 가이드

트윗 감성을 인식하도록 학습된 LLM 아이콘.

Reinforcement Learning from Human Feedback (RLHF)

LLM 파인튜닝 단계별 가이드

트윗 감성을 인식하도록 학습된 LLM이 잘못된 출력을 내는 아이콘.

Reinforcement Learning from Human Feedback (RLHF)

LLM 파인튜닝 단계별 가이드

대규모 데이터셋으로 사전 학습된 트윗 감성 LLM 아이콘.

Reinforcement Learning from Human Feedback (RLHF)

1단계: 사용할 데이터 불러오기

from datasets import load_dataset
import pandas as pd

# `load_dataset` simplifies loading and preprocessing datasets from various sources
# It provides easy access to a wide range of datasets with minimal setup
dataset = load_dataset("mteb/tweet_sentiment_extraction")
df = pd.DataFrame(dataset['train'])

    id               text                                        label   label_text
0   cb774db0d1       I'd have responded, if I were going         1       neutral
1   549e992a42       Sooo SAD I will miss you in San Diego!!!    0       negative
2   08ac60f138       my boss is bullying me...                   0       negative
Reinforcement Learning from Human Feedback (RLHF)

2단계: 사전 학습 모델 선택

from transformers import AutoModelForCausalLM

# AutoModelForCausalLM simplifies loading and switching models
model = AutoModelForCausalLM.from_pretrained("openai-gpt")

 

  • 인과 모델: 이전 토큰이 이후 토큰을 "유도"함
Reinforcement Learning from Human Feedback (RLHF)

3단계: 토크나이저

from transformers import AutoTokenizer

# `AutoTokenizer` loads the correct tokenizer for the specified model
tokenizer = AutoTokenizer.from_pretrained("openai-gpt")
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.resize_token_embeddings(len(tokenizer))

 

  • 패딩: 배치를 동일 길이로 맞춤
Reinforcement Learning from Human Feedback (RLHF)

3단계: 토크나이저

def tokenize_function(examples):
    tokenized = tokenizer(examples["content"], padding="max_length", truncation=True) 
    return tokenized

tokenized_datasets = dataset.map(tokenize_function, batched=True)

 

  • batched 매개변수: 처리 속도 향상
Reinforcement Learning from Human Feedback (RLHF)

4단계: Trainer로 파인튜닝

training_args = TrainingArguments(
   output_dir="test_trainer",
   per_device_train_batch_size=1,
   per_device_eval_batch_size=1,  
   gradient_accumulation_steps=4)

trainer = Trainer(
   model=model,
   args=training_args,
   train_dataset=tokenized_dataset["train"],
   eval_dataset=tokenized_dataset["test"])
trainer.train()
Reinforcement Learning from Human Feedback (RLHF)

연습해 봅시다!

Reinforcement Learning from Human Feedback (RLHF)

Preparing Video For Download...