Agent Final Answer Validation

AI Agents with Hugging Face smolagents

Adel Nehme

VP of AI Curriculum, DataCamp

Why Validation Matters

Agent's answer was not helpful
Customer experience was lost

To avoid this, smolagents lets you validate final answers!

Validating Agent Responses

def check_answer_length(final_answer, agent_memory):
    # Check if the answer is substantial enough
    if len(final_answer) < 200:
        raise Exception("Car recommendation is too brief")
    return True

If final_answer fails rule, raise an exception. Otherwise, return True.

Using Output Validation in Your Agent

car_advisor = CodeAgent(
    tools=[WebSearchTool()],
    model=InferenceClientModel(),
    final_answer_checks=[check_answer_length],
    verbosity_level=0
)

Run check_answer_length validation before responding.
Retry automatically based on exception message defined in the function.

Meta-Evaluation: Using AI to Validate AI

validation_prompt = """
Reasoning process: {}

Agent's final answer: {}

Does the final answer logically follow
from the reasoning and solve the user's 
question? 

Respond only TRUE or FALSE. 
No other text.
"""

Validating Reasoning with a Meta-Evaluator

def check_reasoning_accuracy(final_answer, agent_memory):
    evaluator_model = InferenceClientModel()
    reasoning_steps = agent_memory.get_succinct_steps()
    final_prompt = validation_prompt.format(reasoning_steps, final_answer)

    message = ChatMessage(role='user', content=final_prompt)
    evaluation = evaluator_model([message])

    if evaluation.content == "FALSE":
        raise Exception("The agent's reasoning process contains logical errors")
    else:
        return True

Combining Multiple Validations

car_advisor = CodeAgent(
    tools=[WebSearchTool()],
    model=InferenceClientModel(),
    final_answer_checks=[check_answer_length, check_reasoning_accuracy],
    verbosity_level=0
)

More likely to catch and correct errors before the user ever sees them!

Designing Intelligent Systems

Let's practice!

AI Agents with Hugging Face smolagents