Serving the model

End-to-End Machine Learning

Joshua Stapleton

Machine Learning Engineer

Model-as-a-service

  • Stakeholders / users access model over internet

  • Surface model through portal

    • Users post queries / data
    • Receive diagnosis / predictions

  • Concerns:

    • Rural clinic / no internet access
    • Highly secure environment / sensitive data

Portal symbol showing model as a service

End-to-End Machine Learning

On-device serving

Integrated serving architectures

  • Edge-computing
  • Helpful for unreliable internet cases

Gears in head showing on-device computation

End-to-End Machine Learning

Pros and cons of on-device serving

Pros:

  • Lower latency
  • Security
  • Applications for remote / disconnected areas

Cons:

  • Resource constraints
  • Model updates
  • Monitoring
End-to-End Machine Learning

Implementation strategies

  • Pruning
  • Transfer Learning
  • Use Dedicated Frameworks

Icon showing strategy

End-to-End Machine Learning

Let's practice!

End-to-End Machine Learning

Preparing Video For Download...