Documentloaders integreren

LLM-toepassingen ontwikkelen met LangChain

Jonathan Bennion

AI Engineer & LangChain Contributor

Retrieval Augmented Generation (RAG)

Gebruik embeddings om relevante info te retrieven en in de prompt te verwerken

Een typische RAG-workflow.

Stappen voor RAG-ontwikkeling

De algemene RAG-workflow: een documentloader, een documentsplitser, en opslag + ophalen.

LangChain-documentloaders

Klassen om documenten te laden en configureren voor systeemintegratie
Documentloaders voor gangbare bestandstypen: .pdf, .csv
Loaders van derden: S3, .ipynb, .wav

¹ https://python.langchain.com/docs/integrations/document_loaders

PDF-documentloader

Vereist installatie van het pypdf-pakket: pip install pypdf

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("path/to/file/attention_is_all_you_need.pdf")


data = loader.load()

print(data[0])

Document(page_content='Provided proper attribution is provided, Google hereby grants 
permission to\nreproduce the tables and figures in this paper solely for use in [...]

CSV-documentloader

from langchain_community.document_loaders.csv_loader import CSVLoader


loader = CSVLoader('fifa_countries_audience.csv')


data = loader.load()

print(data[0])

Document(page_content='country: United States\nconfederation: CONCACAF\npopulation_share: [...]

HTML-documentloader

Vereist installatie van het unstructured-pakket: pip install unstructured

from langchain_community.document_loaders import UnstructuredHTMLLoader


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


print(data[0])

print(data[0].metadata)

page_content="To search this site, enter a search term\n\nSearch\n\nExecutive Order on the Safe, Secure,
and Trustworthy Development and Use of Artificial Intelligence\n\nHome\n\nBriefing Room\n\nPresidential
Actions\n\nBy the authority vested in me as President by the Constitution and the laws of the United
States of America, it is hereby ordered as follows: ..."


{'source': 'white_house_executive_order_nov_2023.html'}

Laten we oefenen!

LLM-toepassingen ontwikkelen met LangChain