为检索拆分外部数据

使用 LangChain 开发 LLM 应用

Jonathan Bennion

AI Engineer & LangChain Contributor

RAG 开发步骤

通用 RAG 流程：文档加载器、文档拆分器，以及存储与检索过程。

  • 文档拆分：将文档拆成_块_
  • 拆分以适配 LLM 的上下文窗口
使用 LangChain 开发 LLM 应用

思考如何拆分…

《Attention is All You Need》论文引言的第一段。

第 1 行：

Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks

第 2 行：

in particular, have been firmly established as state of the art approaches in sequence modeling and
1 https://arxiv.org/abs/1706.03762
使用 LangChain 开发 LLM 应用

块重叠（chunk overlap）

将该引言段落拆成两个块并设置重叠的示意图。

使用 LangChain 开发 LLM 应用

最佳的文档拆分策略？

将单词"context"按字符拆块。

 

  1. CharacterTextSplitter
  2. RecursiveCharacterTextSplitter
  3. 还有更多
1 维基百科公有领域
使用 LangChain 开发 LLM 应用
quote = '''One machine can do the work of fifty ordinary humans.\nNo machine can do
the work of one extraordinary human.'''

len(quote)

103

chunk_size = 24
chunk_overlap = 3
1 埃尔伯特·哈伯德
使用 LangChain 开发 LLM 应用
from langchain_text_splitters import CharacterTextSplitter


ct_splitter = CharacterTextSplitter(
    separator='.',
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = ct_splitter.split_text(quote) 
print(docs)

print([len(doc) for doc in docs])

['One machine can do the work of fifty ordinary humans',
 'No machine can do the work of one extraordinary human']

[52, 53]
  • 基于分隔符拆分以使长度 < chunk_size，但不一定总能成功！
使用 LangChain 开发 LLM 应用
from langchain_text_splitters import RecursiveCharacterTextSplitter


rc_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", " ", ""],
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap)


docs = rc_splitter.split_text(quote) 
print(docs)
使用 LangChain 开发 LLM 应用

RecursiveCharacterTextSplitter

  • separators=["\n\n", "\n", " ", ""]
['One machine can do the',
 'work of fifty ordinary',
 'humans.',
 'No machine can do the',
 'work of one',
 'extraordinary human.']
  1. 先按段落拆分："\n\n"
  2. 再按句子："\n"
  3. 再按词：" "
使用 LangChain 开发 LLM 应用

结合 HTML 的 RecursiveCharacterTextSplitter

from langchain_community.document_loaders import UnstructuredHTMLLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


rc_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size,
    chunk_overlap=chunk_overlap,
    separators=['.'])


docs = rc_splitter.split_documents(data) 
print(docs[0])

Document(page_content="To search this site, enter a search term [...]
使用 LangChain 开发 LLM 应用

Vamos praticar!

使用 LangChain 开发 LLM 应用

Preparing Video For Download...