Optimizing AI for speed, cost and quality

AI-Assisted Coding for Developers

Francesca Donadoni

AI Curriculum Manager, DataCamp

Metrics

Latency (response time): How long it takes for the model to generate a response

Metrics

Latency (response time): How long it takes for the model to generate a response

$$

Token cost (per 1M tokens): Monetary expense of using an AI model

Metrics

Latency (response time): How long it takes for the model to generate a response

$$

Token cost (per 1M tokens): Monetary expense of using an AI model

$$

Quality (pass rate on tests/coverage): How often the generated code works as intended

Model benchmarking

$$

¹ https://github.com/openai/human-eval

Model benchmarking

$$

¹ https://github.com/bigcode-project/bigcodebench

Model benchmarking

$$

¹ https://github.com/SWE-bench/SWE-bench

Model benchmarking

$$

¹ https://github.com/JohnnyPeng18/Coffe

Prompt versioning

Prompt versioning

Prompt versioning

Prompt caching

Caching works by storing:
- Prompt
- Input
- Model
- Temperature

Let's practice!

AI-Assisted Coding for Developers

Preparing Video For Download...