Moderation

Entwicklung von KI-Systemen mit der OpenAI-API

Francesca Donadoni

Curriculum Manager, DataCamp

Moderation in der OpenAI API verstehen

Moderation: Prozess, Eingaben zu prüfen, ob sie gegen festgelegte Richtlinien oder Policies verstoßen

Moderation in der OpenAI API verstehen

Ein Diagramm: Die OpenAI-Moderations-API liest eine Nutzernachricht und gibt eine Liste berücksichtigter bösartiger Kategorien aus

Inhalte moderieren

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

Moderation im Kontext

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

Prompt Injection

Eine Frau nutzt einen Chatbot; ein bösartiger Prompt wird injiziert

Prompt Injection

Begrenze die Textmenge im Prompt
Begrenze die Anzahl der Ausgabe-Tokens
Verwende vorausgewählten Inhalt als validierten Input und Output

Schutzmaßnahmen hinzufügen

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there’s only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

Schutzmaßnahmen hinzufügen

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Lass uns üben!

Entwicklung von KI-Systemen mit der OpenAI-API