RAGASでGraph RAGを評価する

Graph RAG with LangChain and Neo4j

Adam Cowley

Manager, Developer Education at Neo4j

Text-to-Cypher

 

結果はプロンプトに挿入され、LLM送信の準備が整う

ベクター検索

 

レビューはスクリーンネーム @emileifrem の Person によって投稿された

Graph RAG with LangChain and Neo4j

Graph RAGの評価

 

  1. time でエンドツーエンド生成を評価
  2. tiktoken でトークン使用量とコストを評価
  3. ragas で出力品質を評価
    • Context precision
    • Noise sensitivity

 

パフォーマンスのトレードオフ

1 GPT-4oで生成した画像
Graph RAG with LangChain and Neo4j

Noise Sensitivity（ノイズ感受性）

  • 取得文書内の「無関係情報の量」を測定
  • ベクター検索のみでは高くなりがち
  • 関連情報に基づくスコアを返す
from ragas.metrics import NoiseSensitivity


metric = NoiseSensitivity(
    llm=evaluator_llm, 
    mode="irrelevant"
)
1 https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity/
Graph RAG with LangChain and Neo4j

Context Precision（文脈精度）

  • 取得文書内の「関連チャンクの割合」を測定
  • スコアが高いほど、取得情報の関連性が高い
from ragas.metrics import LLMContextPrecisionWithReference


metric = LLMContextPrecisionWithReference(
    llm=evaluator_llm, 
)
1 https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/#llm-based-context-precision
Graph RAG with LangChain and Neo4j

Context Precision（文脈精度）

  • 取得文書内の「関連チャンクの割合」を測定
  • スコアが高いほど、取得情報の関連性が高い
from ragas.metrics import LLMContextPrecisionWithoutReference


metric = LLMContextPrecisionWithoutReference(
    llm=evaluator_llm, 
)
Graph RAG with LangChain and Neo4j

Text-to-Cypher の結果構造

cypher_result = {

    "user_input": "Who is Romeo's love?",

    "response": "Romeo loves Juliet",

    "retrieved_contexts": [
        {
            "source": "Romeo",
            "target": "Juliet",
            "relationship": "LOVES",
            "sentiment": 0.9837
        },
    ]

}
Graph RAG with LangChain and Neo4j

Text-to-Cypher の結果構造

cypher_result = {
    "user_input": "Who is Romeo's love?",
    "response": "Romeo loves Juliet",
    "retrieved_contexts": [
        json.dumps({
            "source": "Romeo",
            "target": "Juliet",
            "relationship": "LOVES",
            "sentiment": 0.9837
        }),
    ]
}
Graph RAG with LangChain and Neo4j

ベクターのみの結果構造

vector_result = { 
    "user_input": "Who is Romeo's love?",
    "response": "Romeo loves Juliet",
    "retrieved_contexts": [

        "But, soft! what light through yonder window breaks?..."
        "O, she doth teach the forches to burn bright!"

    ]
}
Graph RAG with LangChain and Neo4j

ハイブリッドの結果構造

hybrid_result = {
    "user_input": "Who is Romeo's love?",
    "response": "Romeo loves Juliet",

    "retrieved_contexts": [

        json.dumps({
            "page_content": "But, soft! what light through yonder window breaks? ...",

            "metadata": {

              "act": 2, "scene": 2, "spoken_to": "Juliet"

            }

        }),

        # ...

    ]
}
Graph RAG with LangChain and Neo4j

評価データセットの作成

cypher_result = {
    "user_input": "Who is Romeo's love?",
    "retrieved_contexts": [
        json.dumps({
            "source": "Romeo",
            "target": "Juliet",
            "relationship": "LOVES",
            "sentiment": 0.9837
        }),
    ]
}


cypher_dataset = EvaluationDataset.from_list([cypher_result])
Graph RAG with LangChain and Neo4j

評価用LLMの選定

from langchain.chat_models import init_chat_model

# 評価用のLLMを選択
llm = init_chat_model(
    "gpt-4o-mini",
    model_provider="openai",
    api_key="...",
    temperature=0
)


# LangchainLLMWrapper でラップ
from ragas.llms import LangchainLLMWrapper
evaluator_llm = LangchainLLMWrapper(llm)
Graph RAG with LangChain and Neo4j

応答の評価

from ragas import evaluate, EvaluationDataset
from ragas.metrics import LLMContextPrecisionWithoutReference, NoiseSensitivity

cypher_scores = evaluate(

    dataset=cypher_dataset,

    metrics=[

        LLMContextPrecisionWithoutReference(llm=evaluator_llm),

        NoiseSensitivity(llm=evaluator_llm, mode="irrelevant")

    ]

)

{'llm_context_precision_without_reference': 1.0000,
'noise_sensitivity(mode=irrelevant)': 0.0000}
Graph RAG with LangChain and Neo4j

Passons à la pratique !

Graph RAG with LangChain and Neo4j

Preparing Video For Download...