Splitting external data for retrieval

Developing LLM Applications with LangChain

Jonathan Bennion

AI Engineer & LangChain Contributor

RAG development steps

The general RAG workflow: a document loader, a document splitter, and the storage and retrieval process.

Document splitting: split document into chunks
Break documents up to fit within an LLM's context window

Thinking about splitting...

The first paragraph from the introduction of the Attention is All You Need paper.

Line 1:

Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks

Line 2:

in particular, have been firmly established as state of the art approaches in sequence modeling and

¹ https://arxiv.org/abs/1706.03762

Chunk overlap

The first paragraph from the introduction of the Attention is All You Need paper split into two chunks with a chunk overlap.

What is the best document splitting strategy?

The word "context" chunked into individual letters.

CharacterTextSplitter
RecursiveCharacterTextSplitter
Many others

¹ Wikipedia Commons

quote = '''One machine can do the work of fifty ordinary humans.\nNo machine can do
the work of one extraordinary human.'''

len(quote)

chunk_size = 24
chunk_overlap = 3

¹ Elbert Hubbard

from langchain_text_splitters import CharacterTextSplitter


ct_splitter = CharacterTextSplitter(
    separator='.',
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = ct_splitter.split_text(quote) 
print(docs)

print([len(doc) for doc in docs])

['One machine can do the work of fifty ordinary humans',
 'No machine can do the work of one extraordinary human']

[52, 53]

Split on separator so < chunk_size, but may not always succeed!

from langchain_text_splitters import RecursiveCharacterTextSplitter


rc_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", " ", ""],
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = rc_splitter.split_text(quote) 
print(docs)

RecursiveCharacterTextSplitter

separators=["\n\n", "\n", " ", ""]

['One machine can do the',
 'work of fifty ordinary',
 'humans.',
 'No machine can do the',
 'work of one',
 'extraordinary human.']

Try splitting by paragraph: "\n\n"
Try splitting by sentence: "\n"
Try splitting by words: " "

RecursiveCharacterTextSplitter with HTML

from langchain_community.document_loaders import UnstructuredHTMLLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


rc_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap,
    separators=['.'])


docs = rc_splitter.split_documents(data) 
print(docs[0])

Document(page_content="To search this site, enter a search term [...]

Let's practice!

Developing LLM Applications with LangChain