Monitoring ML services

MLOps Deployment and Life Cycling

Nemanja Radojkovic

Senior Machine Learning Engineer

Maintaining quality

Paying customers == expectations of quality
Quality assurance starts with quality control
Monitoring

Performance indicators

Fundamental health indicators

Service up and running?
Number of requests in time?
Latency distribution?

Ultimate quality metric

Predictive performance

How do ML models deteriorate?

simple classifier

learned boundary

trained model

world changing

change 2

moved boundary

concept drift

not changing is degrading

how to detect

the catch

by use case

never free

use what you have

input feature changes

covariate shift

input monitoring limitations

output monitoring

Let's practice!

MLOps Deployment and Life Cycling

Preparing Video For Download...