審核

使用 OpenAI API 開發 AI 系統

Francesca Donadoni

Curriculum Manager, DataCamp

了解 OpenAI API 的審核機制

審核（Moderation）：分析輸入以判斷是否含有違反既定政策或指引的內容

了解 OpenAI API 的審核機制

一張圖解：OpenAI 審核 API 讀取使用者訊息為輸入，並回傳所考量的惡意內容類別清單

內容審核

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

情境化的審核

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

提示注入（Prompt injection）

一名女性在使用聊天機器人，畫面被注入惡意提示

提示注入（Prompt injection）

在提示中限制可含的文字量
限制產生的輸出權杖（tokens）數量
以預先選定的內容作為已驗證的輸入與輸出

加入護欄（Guardrails）

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

加入護欄（Guardrails）

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

一起來練習吧！

使用 OpenAI API 開發 AI 系統