Is your deployed AI system successful?

Understanding Artificial Intelligence

Maarten Van den Broeck

Senior Content Developer at DataCamp

When to measure success?

MLOps methodology lifecycle

When to measure success?

Measuring success across the MLOps lifecycle

Measuring performance offline - accuracy

Training and validation data

Measuring performance offline - accuracy

Training a model

Measuring performance offline - accuracy

Validating a model with test examples

Measuring performance offline - accuracy

Comparing predictions to actual labels

Measuring performance offline - accuracy

Calculating accuracy

Beyond accuracy - error and other metrics

Error metric in regression models

Metrics for search and recommendation engines: ranking quality -relevance of ranking items to the user-, diversity in search results or recommendations, etc.

Ranking of recommended products

Measuring success in production

Model degradation: the measured metric value gets worse over the time

Model degradation

Business metrics: Key Performance Indicators (KPIs)
- Indicator of performance and progress of organization objectives

Risks: what could possibly go wrong?

Possible risks include:

Data bias
Lack of transparency
Ethical concerns
Dubious system reliability
Vulnerability to cyber threats

Proof-of-Concept (PoC):

Pilot demonstrator to validate feasibility and potential value + early risk identification

Let's practice!

Understanding Artificial Intelligence