优化文档检索

使用 LangChain 的 Retrieval Augmented Generation (RAG)

Meri Nova

Machine Learning Engineer

把 R 放进 RAG……

从向量数据库检索文档并返回应用以生成响应。

稠密

将块编码为一个含有非零分量的向量

一个向量空间，类似术语聚在一起：Large Language Model、AI、Machine Learning。

优点：捕捉语义
缺点：计算成本高

稠密

将块编码为一个含有非零分量的向量

嵌入在向量空间中的术语，更相似的术语距离更近。

优点：捕捉语义
缺点：计算成本高

稀疏

使用词匹配编码，分量多为零

包含特定术语的文档，其中包含最多术语的文档被高亮。

优点：精确、可解释、处理生僻词
缺点：泛化能力

稀疏检索方法

TF-IDF：用让文档独特的词来编码文档

包含特定术语的文档，其中包含最多术语的文档被高亮。

BM25：降低高频词对编码的"饱和"影响

BM25 检索

from langchain_community.retrievers import BM25Retriever


chunks = [
    "Python was created by Guido van Rossum and released in 1991.",
    "Python is a popular language for machine learning (ML).",
    "The PyTorch library is a popular Python library for AI and ML."
]


bm25_retriever = BM25Retriever.from_texts(chunks, k=3)

BM25 检索

results = bm25_retriever.invoke("When was Python created?")
print("Most Relevant Document:")
print(results[0].page_content)

Most Relevant Document:
Python was created by Guido van Rossum and released in 1991.

Python was created by Guido van Rossum and released in 1991."
"Python is a popular language for machine learning (ML)."
"The PyTorch library is a popular Python library for AI/ML."

RAG 中的 BM25

retriever = BM25Retriever.from_documents(
    documents=chunks, 
    k=5
)


chain = ({"context": retriever, "question": RunnablePassthrough()}
         | prompt
         | llm
         | StrOutputParser()
)

¹ https://www.datacamp.com/blog/what-is-retrieval-augmented-generation-rag

RAG 中的 BM25

print(chain.invoke("How can LLM hallucination impact a RAG application?"))

The RAG application may generate responses that are off-topic or inaccurate.

Passons à la pratique !

使用 LangChain 的 Retrieval Augmented Generation (RAG)