Moderering

Utveckla AI-system med OpenAI API

Francesca Donadoni

Curriculum Manager, DataCamp

Moderering i OpenAI API

Moderering: processen att analysera indata för att avgöra om det innehåller material som bryter mot fördefinierade policyer eller riktlinjer

Moderering i OpenAI API

Ett diagram med ett inmatat användarmeddelande som läses av OpenAI:s moderings-API och som svar producerar en lista över de skadliga innehållskategorier som beaktas

Moderera innehåll

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

Moderering i kontext

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

Promptinjektion

En kvinna som använder en chattbot med en skadlig prompt som injiceras

Promptinjektion

Begränsa mängden text i promptar
Begränsa antalet genererade utdatatokens
Använda förhandsvalt innehåll som validerade in- och utdata

Lägga till skyddsmekanismer

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

Lägga till skyddsmekanismer

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Nu kör vi en övning!

Utveckla AI-system med OpenAI API