Vector databases for embedding systems

Introduction to Embeddings with the OpenAI API

Emmanuel Pire

Senior Software Engineer, DataCamp

Limitations of the current approach

  • Loading all the embeddings into memory (1536 floats ~ 13kB/embedding)
  • Recalculated embeddings for each new query
  • Calculating cosine distances for every embedding and sorting is slow and scales linearly

A diagram showing how the documents and query are embedded with every query with no storage solution.

Introduction to Embeddings with the OpenAI API

Vector databases

  • Embedded documents are stored and queried from the vector database

A diagram showing how a vector database is used to store document embeddings and returned results based on an embedded query back to the user.

Introduction to Embeddings with the OpenAI API

 

NoSQL Database
  • More flexible structure that allows for faster querying

Examples of NoSQL database schemas, including key:value, document, and graph databases.

 

SQL/Relational Database
  • Structured data into tables, rows, and columns

Tables with columns and rows in a SQL, or relational, database.

Introduction to Embeddings with the OpenAI API

Components to store

 

  • Embeddings
  • Source texts
  • Metadata
    • IDs and references
    • Additional data useful for filtering results

 

Top tip: Don't store the source text as metadata!

A vector database icon.

Introduction to Embeddings with the OpenAI API

The vector database landscape

Many of the most popular vector database solutions, sorted by whether they are closed source or open source, and whether they are dedicated vectors databases or not.

1 Image Credit: Yingjun Wu
Introduction to Embeddings with the OpenAI API

Which solution is best?

 

  • Database management:
    • Managed → more expensive but lowers workload
    • Self-managed → cheaper but requires time and expertise
  • Open source or commercial?
    • Open source → flexible and cost-effective
    • Commercial → better support, more advanced features, and compliance

 

  • Data models: does the type of data lend itself to a particular database type?
  • Specific features: does your use case depend on specific functionality, such as multi-modal storage?

Chroma logo.

Introduction to Embeddings with the OpenAI API

Let's practice!

Introduction to Embeddings with the OpenAI API

Preparing Video For Download...