文本切分、嵌入与向量存储

使用 LangChain 的 Retrieval Augmented Generation (RAG)

Meri Nova

Machine Learning Engineer

为检索做数据准备

正在加载文档。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

为检索做数据准备

文档被切分。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

为检索做数据准备

文档块被嵌入。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

为检索做数据准备

文档块被存储。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

为检索做数据准备

在 RAG 开发流程中高亮显示了切分步骤。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

chunk_size

一个箭头展示理想块大小在中间；块太大会导致检索变慢和难以理解，块太小则上下文不足。

chunk_overlap

  • 在块的边界之外包含信息

两个块之间高亮显示的重叠区域。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

CharacterTextSplitter

from langchain_text_splitters import CharacterTextSplitter


text = """Machine learning is a fascinating field.\n\nIt involves algorithms and models that
can learn from data. These models can then make predictions or decisions without being
explicitly programmed to perform the task.\nThis capability is increasingly valuable in
various industries, from finance to healthcare.\n\nThere are many types of machine learning,
including supervised, unsupervised, and reinforcement learning.\nEach type has its own
strengths and applications."""


text_splitter = CharacterTextSplitter(
    separator="\n\n",
    chunk_size=100,  
    chunk_overlap=10  
)
使用 LangChain 的 Retrieval Augmented Generation (RAG)

CharacterTextSplitter

chunks = text_splitter.split_text(text)

print(chunks)
print([len(chunk) for chunk in chunks])

['Machine learning is a fascinating field.',
 'It involves algorithms and models that can learn from data. These models can...',
 'There are many types of machine learning, including supervised, unsupervised...']

[40, 260, 155]
  • 可能缺少上下文
  • 可能超过设定的 chunk_size
使用 LangChain 的 Retrieval Augmented Generation (RAG)

RecursiveCharacterTextSplitter

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

    separators=["\n\n", "\n", " ", ""],

    chunk_size=100,
    chunk_overlap=10

)
使用 LangChain 的 Retrieval Augmented Generation (RAG)

RecursiveCharacterTextSplitter

chunks = splitter.split_text(text)

print(chunks)
print([len(chunk) for chunk in chunks])

['Machine learning is a fascinating field.',
 'It involves algorithms and models that can learn from data. These models ...',
 'or decisions without being explicitly programmed to perform the task.',
 'This capability is increasingly valuable in various industries, from ...',
 'There are many types of machine learning, including supervised, ...',
 'learning.',
 'Each type has its own strengths and applications.']
[40, 98, 69, 91, 95, 9, 49]
使用 LangChain 的 Retrieval Augmented Generation (RAG)

切分文档

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("research_paper.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)


chunks = splitter.split_documents(documents)
使用 LangChain 的 Retrieval Augmented Generation (RAG)

切分文档

print(chunks)

print([len(chunk.page_content) for chunk in chunks])

[Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...'),
 Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...'),
 Document(metadata={'source': 'Rag Paper.pdf', 'page': 0}, page_content='...')]

[928, 946, 921,...]
使用 LangChain 的 Retrieval Augmented Generation (RAG)

嵌入与存储

突出显示了嵌入与存储步骤。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

什么是嵌入？

一句话被传入嵌入模型。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

什么是嵌入？

嵌入模型将文本表示为数值向量。

使用 LangChain 的 Retrieval Augmented Generation (RAG)

什么是嵌入？

what_are_embeddings3.jpg

使用 LangChain 的 Retrieval Augmented Generation (RAG)

什么是嵌入？

what_are_embeddings4.jpg

使用 LangChain 的 Retrieval Augmented Generation (RAG)

对块进行嵌入并存储

  • 使用 OpenAI 和 ChromaDB 进行嵌入与存储
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embedding_model = OpenAIEmbeddings(
    api_key=openai_api_key,
    model="text-embedding-3-small"
)


vector_store = Chroma.from_documents(
    documents=chunks, 
    embedding=embedding_model
)
使用 LangChain 的 Retrieval Augmented Generation (RAG)

Passons à la pratique !

使用 LangChain 的 Retrieval Augmented Generation (RAG)

Preparing Video For Download...