Mengintegrasikan pemuat dokumen

Mengembangkan Aplikasi LLM dengan LangChain

Jonathan Bennion

AI Engineer & LangChain Contributor

Retrieval Augmented Generation (RAG)

Gunakan embedding untuk mengambil informasi relevan dan mengintegrasikannya ke prompt

Alur kerja RAG khas.

Langkah pengembangan RAG

Alur umum RAG: pemuat dokumen, pemecah dokumen, serta proses penyimpanan dan pengambilan.

Pemuat dokumen LangChain

Kelas untuk memuat dan mengonfigurasi dokumen agar siap diintegrasikan
Pemuat untuk tipe umum: .pdf, .csv
Pemuat pihak ketiga: S3, .ipynb, .wav

¹ https://python.langchain.com/docs/integrations/document_loaders

Pemuat dokumen PDF

Memerlukan instalasi paket pypdf: pip install pypdf

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("path/to/file/attention_is_all_you_need.pdf")


data = loader.load()

print(data[0])

Document(page_content='Provided proper attribution is provided, Google hereby grants 
permission to\nreproduce the tables and figures in this paper solely for use in [...]

Pemuat dokumen CSV

from langchain_community.document_loaders.csv_loader import CSVLoader


loader = CSVLoader('fifa_countries_audience.csv')


data = loader.load()

print(data[0])

Document(page_content='country: United States\nconfederation: CONCACAF\npopulation_share: [...]

Pemuat dokumen HTML

Memerlukan instalasi paket unstructured: pip install unstructured

from langchain_community.document_loaders import UnstructuredHTMLLoader


loader = UnstructuredHTMLLoader("white_house_executive_order_nov_2023.html")
data = loader.load()


print(data[0])

print(data[0].metadata)

page_content="To search this site, enter a search term\n\nSearch\n\nExecutive Order on the Safe, Secure,
and Trustworthy Development and Use of Artificial Intelligence\n\nHome\n\nBriefing Room\n\nPresidential
Actions\n\nBy the authority vested in me as President by the Constitution and the laws of the United
States of America, it is hereby ordered as follows: ..."


{'source': 'white_house_executive_order_nov_2023.html'}

Ayo berlatih!

Mengembangkan Aplikasi LLM dengan LangChain