Moderação

Desenvolvimento de sistemas de IA com a API OpenAI

Francesca Donadoni

Curriculum Manager, DataCamp

Entendendo a moderação na API da OpenAI

Moderação: processo de analisar a entrada para ver se viola políticas ou diretrizes pré-definidas

Entendendo a moderação na API da OpenAI

Um diagrama com uma mensagem do usuário lida pela API de moderação da OpenAI e, como resposta, uma lista das categorias de conteúdo malicioso consideradas

Moderando conteúdo

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

Moderação em contexto

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

Injeção de prompt

Uma mulher usando um chatbot com um prompt malicioso sendo injetado

Injeção de prompt

Limitar a quantidade de texto nos prompts
Limitar o número de tokens de saída gerados
Usar conteúdo pré-selecionado como entrada e saída validadas

Adicionando guardrails

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

Adicionando guardrails

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Vamos praticar!

Desenvolvimento de sistemas de IA com a API OpenAI