Returning structured prediction response

Deploying AI into Production with FastAPI

Matt Eckerle

Software and Data Engineering Leader

Challenges with deploying models

Flowchart showing ML prediction with FastAPI

Accept input data properly
Validate incoming data and handle errors
Make predictions
Return well-structured responses

Defining request structure

from pydantic import BaseModel

class PredictionRequest(BaseModel):
    text: str

class PredictionResponse(BaseModel):

    text: str

    sentiment: str

    confidence: float

Creating the prediction endpoint

@app.post("/predict")
def predict_sentiment(request: PredictionRequest):
    if sentiment_model is None:
        raise HTTPException(
            status_code=503,
            detail="Model not loaded"
        )

    result = sentiment_model(request.text)

    return PredictionResponse(
            text=request.text,
            sentiment=result[0]["label"],
            confidence=result[0]["score"]
        )

Input JSON:

{"text": "This movie was fantastic!"}

Response:

{
    "text": "This movie was fantastic!",
    "sentiment": "POSITIVE",
    "confidence": 0.95
}

Error handling

try:
  result = sentiment_model(request.text)
  return PredictionResponse(
        text=request.text,
        sentiment=result[0]["label"],
        confidence=result[0]["score"]
    )
except Exception:
  raise HTTPException(
        status_code=500,
        detail="Prediction failed"
    )

Response when model fails to predict

{
    "detail": "Prediction failed",
    "status_code": 500
}

Testing the endpoint

# Example request
import requests

response = requests.post(
    "http://localhost:8000/predict",
    json={"text": "Great product!"}
)
print(response.json())

{
    "text": "Great product!",
    "sentiment": "POSITIVE",
    "confidence": 0.998
}

Let's practice!

Deploying AI into Production with FastAPI