Monitoring ML services

Deployment e ciclo di vita in MLOps

Nemanja Radojkovic

Senior Machine Learning Engineer

Maintaining quality

  • Paying customers == expectations of quality
  • Quality assurance starts with quality control
  • Monitoring
Deployment e ciclo di vita in MLOps

Performance indicators

Fundamental health indicators

  • Service up and running?
  • Number of requests in time?
  • Latency distribution?

 

Ultimate quality metric

  • Predictive performance

How do ML models deteriorate?

Deployment e ciclo di vita in MLOps

simple classifier

Deployment e ciclo di vita in MLOps

learned boundary

Deployment e ciclo di vita in MLOps

trained model

Deployment e ciclo di vita in MLOps

world changing

Deployment e ciclo di vita in MLOps

change 2

Deployment e ciclo di vita in MLOps

moved boundary

Deployment e ciclo di vita in MLOps

concept drift

Deployment e ciclo di vita in MLOps

not changing is degrading

Deployment e ciclo di vita in MLOps

how to detect

Deployment e ciclo di vita in MLOps

Deployment e ciclo di vita in MLOps

the catch

Deployment e ciclo di vita in MLOps

by use case

Deployment e ciclo di vita in MLOps

never free

Deployment e ciclo di vita in MLOps

use what you have

Deployment e ciclo di vita in MLOps

input feature changes

Deployment e ciclo di vita in MLOps

covariate shift

Deployment e ciclo di vita in MLOps

input monitoring limitations

Deployment e ciclo di vita in MLOps

output monitoring

Deployment e ciclo di vita in MLOps

Let's practice!

Deployment e ciclo di vita in MLOps

Preparing Video For Download...