Модерация

Разработка ИИ-систем с помощью OpenAI API

Francesca Donadoni

Curriculum Manager, DataCamp

Модерация в OpenAI API

Модерация: процесс анализа входных данных для выявления содержимого, нарушающего заданные правила или политики

Модерация в OpenAI API

Диаграмма: входное сообщение пользователя обрабатывается API модерации OpenAI, который возвращает ответ со списком категорий нежелательного контента

Модерация контента

moderation_response = client.moderations.create(input="""
...until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until...
""") 

print(moderation_response.results[0].categories.violence)

True

¹ https://ek.explodingkittens.com/how-to-play/exploding-kittens

Модерация в контексте

moderation_response = client.moderations.create(input="""
In the deck of cards are some Exploding Kittens. You play the game by putting the deck face down and taking turns drawing cards until someone draws an Exploding Kitten.
When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
""") 

moderation_response.results[0].categories.violence

False

Внедрение промптов

Женщина использует чат-бот, в который внедряется вредоносный промпт

Внедрение промптов

Ограничение объёма текста в промптах
Ограничение числа генерируемых токенов вывода
Использование заранее проверенного контента в качестве входных и выходных данных

Добавление ограничений

user_request = """
In the deck of cards are some Exploding Kittens. You play the game by putting the 
deck face down and taking turns drawing cards until  someone draws an Exploding 
Kitten. When that happens, that person explodes. They are now dead.
This process continues until there's only 1 player left, who wins the game.
The more cards you draw, the greater your chances of drawing an Exploding Kitten.
"""

messages = [{"role": "system",
             "content": "Your role is to assess whether the user question is 
              allowed or not. The allowed topics are games of chess only. If 
              the topic is allowed, reply with an answer as normal, otherwise
              say 'Apologies, but the topic is not_allowed.'",},
            {"role": "user", "content": user_request},]

Добавление ограничений

response = client.chat.completions.create(
    model="gpt-4o-mini", 
    messages=messages
)

print(response.choices[0].message.content)

Apologies, but the topic is not allowed.

Давайте потренируемся!

Разработка ИИ-систем с помощью OpenAI API