Serving the model

Machine Learning end-to-end

Joshua Stapleton

Machine Learning Engineer

Model-as-a-service

  • Stakeholders / users access model over internet

  • Surface model through portal

    • Users post queries / data
    • Receive diagnosis / predictions

  • Concerns:

    • Rural clinic / no internet access
    • Highly secure environment / sensitive data

Portal symbol showing model as a service

Machine Learning end-to-end

On-device serving

Integrated serving architectures

  • Edge-computing
  • Helpful for unreliable internet cases

Gears in head showing on-device computation

Machine Learning end-to-end

Pros and cons of on-device serving

Pros:

  • Lower latency
  • Security
  • Applications for remote / disconnected areas

Cons:

  • Resource constraints
  • Model updates
  • Monitoring
Machine Learning end-to-end

Implementation strategies

  • Pruning
  • Transfer Learning
  • Use Dedicated Frameworks

Icon showing strategy

Machine Learning end-to-end

Let's practice!

Machine Learning end-to-end

Preparing Video For Download...