Designing reproducible experiments

Developing Machine Learning Models for Production

Sinan Ozdemir

Data Scientist and Author

Reproducible experiments

Ensure accuracy and reliability.
Easier to replicate results.
Allow for better collaboration.
Help reduce the risk of bias in ML models.
Bolster the integrity of the research process.

MLflow

An open-source platform for tracking and managing machine learning experiments. MLflow can be used to:

Create reproducible ML pipelines
Track and manage:
- package dependencies
- code versions
- experiment settings
Allows multiple users to access experiments

mlflow

¹ https://www.databricks.com/blog/2018/06/05/introducing-mlflow-an-open-source-machine-learning-platform.html

Example of using Mlflow

# standard scikit-learn imports
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# new imports from MLflow
import mlflow
import mlflow.sklearn

Example of using MLflow cont.

with mlflow.start_run():  # Start an MLflow run assuming we have data prepared

    # Build and train model
    rf = RandomForestClassifier()
    rf.fit(X_train, y_train)

    # Log parameters and model information
    mlflow.log_param("n_estimators", rf.n_estimators)
    mlflow.sklearn.log_model(rf, "model")

    y_pred = rf.predict(X_test)  # Evaluate model
    accuracy = accuracy_score(y_test, y_pred)
    mlflow.log_metric("accuracy", accuracy)  # log the test accuracy metric

Tracking code

Logging code versions and changes with MLflow
Comparing different versions of code
Identifying which version of the code was used to produce a given set of results
Easily reproducing experiments
Making it easy to debug and troubleshoot code

Model registries

Centralized repository of models and their metadata
MLflow can be used to log, store and compare different versions of models
Reproducing entire ML pipelines
Used for comparing models
Ensuring accuracy and reliability of models

storage

Experiment reproducibility

Tracking and logging input data, code, and settings
Reproducing entire ML pipelines
Building trust in the results of machine learning models
Allowing others to verify and build upon the work
Ensuring results are consistent across different runs

Revisiting documentation

Documentation is essential for reproducible ML and should include information about:
- Models' input data.
- The code used to create models.
- Any settings used in an experiment.
- The results of the experiment (which model was chosen, etc)
Documentation should be accessible
Meant to be a full record of an experiment

Let's practice!

Developing Machine Learning Models for Production