Externe data splitsen voor retrieval

LLM-toepassingen ontwikkelen met LangChain

Jonathan Bennion

AI Engineer & LangChain Contributor

RAG-ontwikkelstappen

De algemene RAG-workflow: een document loader, een document splitter, en opslag- en retrievalproces.

Document splitting: document opdelen in chunks
Splits om binnen het contextvenster van een LLM te passen

Nadenken over splitsen...

De eerste alinea uit de introductie van het artikel Attention Is All You Need.

Regel 1:

Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks

Regel 2:

in particular, have been firmly established as state of the art approaches in sequence modeling and

¹ https://arxiv.org/abs/1706.03762

Chunk-overlap

De eerste alinea uit de introductie van het artikel Attention Is All You Need, gesplitst in twee chunks met overlap.

Wat is de beste strategie om te splitsen?

Het woord "context" opgedeeld in losse letters.

CharacterTextSplitter
RecursiveCharacterTextSplitter
Veel andere

¹ Wikipedia Commons

quote = '''One machine can do the work of fifty ordinary humans.\nNo machine can do
the work of one extraordinary human.'''

len(quote)

chunk_size = 24
chunk_overlap = 3

¹ Elbert Hubbard

from langchain_text_splitters import CharacterTextSplitter


ct_splitter = CharacterTextSplitter(
    separator='.',
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = ct_splitter.split_text(quote) 
print(docs)

print([len(doc) for doc in docs])

['One machine can do the work of fifty ordinary humans',
 'No machine can do the work of one extraordinary human']

[52, 53]

Splits op scheidingsteken zodat < chunk_size, maar lukt niet altijd!

from langchain_text_splitters import RecursiveCharacterTextSplitter


rc_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", " ", ""],
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = rc_splitter.split_text(quote) 
print(docs)

RecursiveCharacterTextSplitter

separators=["\n\n", "\n", " ", ""]

['One machine can do the',
 'work of fifty ordinary',
 'humans.',
 'No machine can do the',
 'work of one',
 'extraordinary human.']

Probeer te splitsen op alinea: "\n\n"
Probeer te splitsen op zin: "\n"
Probeer te splitsen op woorden: " "

RecursiveCharacterTextSplitter met HTML

from langchain_community.document_loaders import UnstructuredHTMLLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


rc_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap,
    separators=['.'])


docs = rc_splitter.split_documents(data) 
print(docs[0])

Document(page_content="To search this site, enter a search term [...]

Laten we oefenen!

LLM-toepassingen ontwikkelen met LangChain