Scalability

Developing Machine Learning Models for Production

Sinan Ozdemir

Data Scientist, Entrepreneur, and Author

Compute constraints in scalability

  • CPU, memory, and disk requirements can impact ML model scalability
  • Measure CPU, memory, and disk usage during training and inference
  • Early identification leads to better model scalability

server room

Developing Machine Learning Models for Production

Model complexity and scalability

  • Model complexity affects scalability
  • Balancing complexity and scalability is challenging
  • Strategies for balancing:
    • Feature selection techniques (e.g. Chi-squared, PCA)
    • Model compression techniques (e.g. pruning)

PCA

Principal Component Analysis (PCA)

Developing Machine Learning Models for Production

Velocity of deployments and scalability

  • Rapid deployment and iteration is crucial for relevance and accuracy
  • Strategies
    • continuous integration/deployment - covered in the next lesson
    • online learning - learning with new data in real time

runner

Developing Machine Learning Models for Production

Optimal scaling strategies

  • Trade-offs include cost of serving, re-training, and velocity of deployments
  • Scaling strategies: horizontal scaling, vertical scaling, auto-scaling
    • Horizontal scaling: adding more machines
    • Vertical scaling: increasing machine size
    • Auto-scaling: adjusts number of machines based on workload

scaling

1 https://www.spiceworks.com/tech/cloud/articles/horizontal-vs-vertical-cloud-scaling
Developing Machine Learning Models for Production

Considering your optimal scaling strategy

  • Horizontal scaling uses load balancing, partitioning

  • Horizontal scaling trade-offs: complexity, cost ($)

  • Vertical scaling increases machine size

  • Vertical scaling trade-off: cost ($)

Ideally - auto-scale horizontally and vertically.

scaling

1 https://www.spiceworks.com/tech/cloud/articles/horizontal-vs-vertical-cloud-scaling
Developing Machine Learning Models for Production

Let's practice!

Developing Machine Learning Models for Production

Preparing Video For Download...