Testing a model

End-to-End Machine Learning

Joshua Stapleton

Machine Learning Engineer

Deployment

  • Next step in ML lifecycle
  • After training and evaluation
  • Make model available for use

Deployment phase of machine learning lifecycle

End-to-End Machine Learning

Testing

  • Testing:
    • Model does not crash
    • Returning reasonable outputs at inference time
    • Returning outputs in reasonable time

Deployment phase of machine learning lifecycle

End-to-End Machine Learning

Unittest

  • Testing
    • Flag anomalous / unexpected events.
    • Check model is performing as expected.

 

  • unittest
    • Built-in Python library for test-writing.
    • Test case: covers type of test.
    • Test case method: single test for one aspect of test case.
End-to-End Machine Learning

Unittest usage

import unittest


class TestModelInference(unittest.TestCase):

    def setUp(self):
        self.model = fitted_model
        self.X_test = X_test

    def test_prediction_output_shape(self):
        y_pred = self.model.predict(self.X_test)
        self.assertEqual(y_pred.shape[0], self.X_test.shape[0])


if __name__ == '__main__':
    unittest.main()
End-to-End Machine Learning

Unittest usage (cont.)

 

    def test_input_values(self):
        print("Running test_input_values test case")

        # Get inputs (each row in testing set)
        for input in X_test: 
            for value in input:
                # if value is cholestrol, for example:
                self.assertIn(value, [0, 500])
End-to-End Machine Learning

Testing in Python

Introduction to Testing in Python

 

Testing in Python course cover page

End-to-End Machine Learning

Testing do's and dont's

Best-practices

  • DON'T...
    • Write too many tests
    • Write redundant tests
    • Write tests for highly reliable components.

 

  • DO...
    • Write tests to increase reliability.
    • Write tests to check/manage expectations.
    • Write tests for new functionality.
End-to-End Machine Learning

Testing benefits

Benefits of TDD

  • Confidence in development
    • Stable iteration
    • Less worry about bugs
  • Performance
    • Reliability
    • Production-grade
End-to-End Machine Learning

Let's practice!

End-to-End Machine Learning

Preparing Video For Download...