审核

使用 OpenAI API 构建 AI 系统

Francesca Donadoni

Curriculum Manager, DataCamp

理解 OpenAI API 中的审核

审核（Moderation）：分析输入以判断其是否含有违反既定政策或指南的内容

理解 OpenAI API 中的审核

一个示意图：OpenAI 审核 API 读取用户输入消息，并在响应中给出所考虑的恶意内容类别列表

内容审核

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

情境化审核

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

提示注入

一名女性在使用聊天机器人，提示被注入了恶意内容

提示注入

限制提示中的文本长度
限制生成的输出 tokens 数
使用预选内容作为已验证的输入与输出

添加护栏

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

添加护栏

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Ayo berlatih!

使用 OpenAI API 构建 AI 系统