Deployment

LLMOps Concepts

Max Knobbout, PhD

Applied Scientist, Uber

LLM lifecycle: Deployment

Overview of the LLM application lifecycle phases

LLMOps Concepts

Moving to deployment

 

 

Playful image of cartoon characters dragging a box

 

 

  • No one-size-fits-all!
  • An application may include a chain/agent logic, vector database, LLM, and more
  • Each component needs to be deployed and work together
LLMOps Concepts

Step 1: Choice of hosting

  • Private/public cloud or on-premise hosting
  • Many cloud providers offer solutions for LLM hosting and deployment

Hosting

LLMOps Concepts

Step 2: API design

  • APIs let different software talk to each other
  • Design affects scalability, cost and infrastructure needs
  • Security is crucial, controlled with API keys!

API design

LLMOps Concepts

Step 3: How to run

  • Options:

    1. Containers
    2. Serverless functions
    3. Cloud managed services

  • Advantages/disadvantages like costs, scalability, efficiency and flexibility

Docker logo

Execution architecture

LLMOps Concepts

CI/CD

Continuous Integration (CI):

  1. Source: Retrieve source code
  2. Build: Create a container image containing the code
  3. Test: Perform integration tests
  4. Register: Store the container in a registry

Continuous Deployment (CD):

  1. Retrieve: Retrieve container from registry
  2. Test: Perform deployment tests
  3. Deploy: Deploy container to environments:
    • Staging
    • Production
LLMOps Concepts

Scaling

 

Playful image of cartoon characters building a large scale robot

 

  • LLMs might need specialized GPU hardware.

  • Scaling strategies:

    1. Horizontal: Add more machines
    2. Vertical: Boosting one machine

  • Horizontal for traffic, vertical for reliability and speed

LLMOps Concepts

Let's practice!

LLMOps Concepts

Preparing Video For Download...