Tekstsplitsing, embeddings en vectoropslag

Retrieval Augmented Generation (RAG) met LangChain

Meri Nova

Machine Learning Engineer

Data voorbereiden voor retrieval

Documenten worden geladen.

Data voorbereiden voor retrieval

Documenten worden gesplitst.

Data voorbereiden voor retrieval

Document-chunks worden ge-embed.

Data voorbereiden voor retrieval

Document-chunks worden opgeslagen.

Data voorbereiden voor retrieval

De splitsstap is gemarkeerd in de RAG-ontwikkelworkflow.

chunk_size

Een pijl toont dat de ideale chunk-grootte in het midden ligt; grote chunks kunnen trage opvraging en lastigere interpretatie geven, en kleine chunks hebben te weinig context.

chunk_overlap

Neem info op die over de grens gaat

Twee chunks met een gemarkeerd gebied dat overlapt.

CharacterTextSplitter

from langchain_text_splitters import CharacterTextSplitter


text = """Machine learning is a fascinating field.\n\nIt involves algorithms and models that
can learn from data. These models can then make predictions or decisions without being
explicitly programmed to perform the task.\nThis capability is increasingly valuable in
various industries, from finance to healthcare.\n\nThere are many types of machine learning,
including supervised, unsupervised, and reinforcement learning.\nEach type has its own
strengths and applications."""


text_splitter = CharacterTextSplitter(
    separator="\n\n",
    chunk_size=100,  
    chunk_overlap=10  
)

CharacterTextSplitter

chunks = text_splitter.split_text(text)

print(chunks)
print([len(chunk) for chunk in chunks])

['Machine learning is a fascinating field.',
 'It involves algorithms and models that can learn from data. These models can...',
 'There are many types of machine learning, including supervised, unsupervised...']

[40, 260, 155]

Chunks kunnen context missen
Chunks kunnen groter zijn dan chunk_size

RecursiveCharacterTextSplitter

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

    separators=["\n\n", "\n", " ", ""],

    chunk_size=100,
    chunk_overlap=10

)

RecursiveCharacterTextSplitter

chunks = splitter.split_text(text)

print(chunks)
print([len(chunk) for chunk in chunks])

['Machine learning is a fascinating field.',
 'It involves algorithms and models that can learn from data. These models ...',
 'or decisions without being explicitly programmed to perform the task.',
 'This capability is increasingly valuable in various industries, from ...',
 'There are many types of machine learning, including supervised, ...',
 'learning.',
 'Each type has its own strengths and applications.']
[40, 98, 69, 91, 95, 9, 49]

Documenten splitsen

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("research_paper.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)


chunks = splitter.split_documents(documents)

Documenten splitsen

print(chunks)

print([len(chunk.page_content) for chunk in chunks])

[Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...'),
 Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...'),
 Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...')]

[928, 946, 921,...]

Embeddings en opslag

De stappen embedden en opslaan zijn gemarkeerd.

Wat zijn embeddings?

Een zin wordt naar een embeddingmodel gestuurd.

Wat zijn embeddings?

Het embeddingmodel zet tekst om naar een vector met numerieke waarden.

Wat zijn embeddings?

Chunks embedden en opslaan

Embed en sla op met: OpenAI en ChromaDB

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embedding_model = OpenAIEmbeddings(
    api_key=openai_api_key,
    model="text-embedding-3-small"
)


vector_store = Chroma.from_documents(
    documents=chunks, 
    embedding=embedding_model
)

Laten we oefenen!

Retrieval Augmented Generation (RAG) met LangChain