Memecah data eksternal untuk penelusuran

Mengembangkan Aplikasi LLM dengan LangChain

Jonathan Bennion

AI Engineer & LangChain Contributor

Langkah pengembangan RAG

Alur kerja RAG umum: pemuat dokumen, pemecah dokumen, serta proses penyimpanan dan penelusuran.

Pemecahan dokumen: bagi dokumen menjadi chunk
Pecah dokumen agar muat dalam context window LLM

Memikirkan pemecahan...

Paragraf pertama dari pengantar makalah Attention is All You Need.

Baris 1:

Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks

Baris 2:

in particular, have been firmly established as state of the art approaches in sequence modeling and

¹ https://arxiv.org/abs/1706.03762

Tumpang tindih chunk

Paragraf pertama dari pengantar makalah Attention is All You Need dibagi menjadi dua chunk dengan tumpang tindih chunk.

Strategi pemecahan dokumen yang terbaik?

Kata "context" dipecah menjadi huruf per huruf.

CharacterTextSplitter
RecursiveCharacterTextSplitter
Lainnya

¹ Wikipedia Commons

quote = '''One machine can do the work of fifty ordinary humans.\nNo machine can do
the work of one extraordinary human.'''

len(quote)

chunk_size = 24
chunk_overlap = 3

¹ Elbert Hubbard

from langchain_text_splitters import CharacterTextSplitter


ct_splitter = CharacterTextSplitter(
    separator='.',
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = ct_splitter.split_text(quote) 
print(docs)

print([len(doc) for doc in docs])

['One machine can do the work of fifty ordinary humans',
 'No machine can do the work of one extraordinary human']

[52, 53]

Pecah berdasarkan pemisah agar < chunk_size, tetapi tidak selalu berhasil!

from langchain_text_splitters import RecursiveCharacterTextSplitter


rc_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", " ", ""],
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = rc_splitter.split_text(quote) 
print(docs)

RecursiveCharacterTextSplitter

separators=["\n\n", "\n", " ", ""]

['One machine can do the',
 'work of fifty ordinary',
 'humans.',
 'No machine can do the',
 'work of one',
 'extraordinary human.']

Coba pecah per paragraf: "\n\n"
Coba pecah per kalimat: "\n"
Coba pecah per kata: " "

RecursiveCharacterTextSplitter untuk HTML

from langchain_community.document_loaders import UnstructuredHTMLLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


rc_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap,
    separators=['.'])


docs = rc_splitter.split_documents(data) 
print(docs[0])

Document(page_content="To search this site, enter a search term [...]

Ayo berlatih!

Mengembangkan Aplikasi LLM dengan LangChain