Moderazione

Sviluppare sistemi di AI con l'API di OpenAI

Francesca Donadoni

Curriculum Manager, DataCamp

Capire la moderazione nell'API OpenAI

Moderazione: analisi dell'input per verificare se viola policy o linee guida

Capire la moderazione nell'API OpenAI

Schema: un messaggio utente letto dalla OpenAI moderation API con, in risposta, l'elenco delle categorie di contenuti malevoli considerate

Moderare i contenuti

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

Moderazione in contesto

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

Prompt injection

Una donna usa un chatbot con un prompt malevolo iniettato

Prompt injection

Limitare la quantità di testo nei prompt
Limitare il numero di token di output generati
Usare contenuti preselezionati come input e output validati

Aggiungere guardrail

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

Aggiungere guardrail

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Ayo berlatih!

Sviluppare sistemi di AI con l'API di OpenAI