모더레이션

OpenAI API로 AI 시스템 개발하기

Francesca Donadoni

Curriculum Manager, DataCamp

OpenAI API의 모더레이션 이해

모더레이션: 입력 내용이 사전 정의된 정책이나 지침을 위반하는지 분석하는 프로세스

OpenAI API의 모더레이션 이해

사용자 입력 메시지를 OpenAI 모더레이션 API가 읽고 고려되는 악성 콘텐츠 카테고리 목록을 응답으로 출력하는 다이어그램

콘텐츠 모더레이션

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

맥락 속 모더레이션

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

프롬프트 인젝션

악성 프롬프트가 삽입된 챗봇을 사용하는 여성

프롬프트 인젝션

프롬프트 내 텍스트 양 제한
생성되는 출력 토큰 수 제한
사전 선택된 콘텐츠를 검증된 입출력으로 사용

가드레일 추가

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

가드레일 추가

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

연습해 봅시다!

OpenAI API로 AI 시스템 개발하기