spaCy で意味的類似度を測る

spaCyで学ぶNatural Language Processing

Azadeh Mobasher

Principal Data Scientist

意味的類似度の手法

テキストを分析し、類似性を特定
テキストを事前定義カテゴリに分類、または関連テキストを検出
類似度スコアは2つのテキストの近さを測定

What is the cheapest flight from Boston to Seattle?
Which airline serves Denver, Pittsburgh and Atlanta?
What kinds of planes are used by American Airlines?

類似度スコア

テキスト上で定義される指標
類似度の測定に コサイン類似度 と 単語ベクトル を使用
コサイン類似度は 0〜1 の値

コサイン類似度とベクトル

トークンの類似度

spaCy は Token オブジェクト間の類似度を計算

nlp = spacy.load("en_core_web_md")
doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

token1 = doc1[2]
token2 = doc2[4]
print(f"Similarity between {token1} and {token2} = ", round(token1.similarity(token2), 3))

>>> Similarity between pizza and pasta =  0.685

スパンの類似度

spaCy は2つの Span オブジェクトの意味的類似度を計算

doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

span1 = doc1[1:]
span2 = doc2[1:]

print(f"Similarity between \"{span1}\" and \"{span2}\" = ",
        round(span1.similarity(span2), 3))

>>> Similarity between "eat pizza" and "like to eat pasta" =  0.588

print(f"Similarity between \"{doc1[1:]}\" and \"{doc2[3:]}\" = ",
        round(doc1[1:].similarity(doc2[3:]), 3))

>>> Similarity between "eat pizza" and "eat pasta" =  0.936

Doc の類似度

spaCy は2つのドキュメント間の類似度を計算

nlp = spacy.load("en_core_web_md")

doc1 = nlp("I like to play basketball")
doc2 = nlp("I love to play basketball")
print("Similarity score :", round(doc1.similarity(doc2), 3))

>>> Similarity score : 0.975

コサイン類似度が高いほど意味的に近い内容
Doc ベクトルは既定で単語ベクトルの平均

文の類似度

spaCy は指定したキーワードに関連する内容を検出
例: price に近い顧客質問を検索

sentences = nlp("What is the cheapest flight from Boston to Seattle? 
                 Which airline serves Denver, Pittsburgh and Atlanta? 
                 What kinds of planes are used by American Airlines?")

keyword = nlp("price")

for i, sentence in enumerate(sentences.sents):
    print(f"Similarity score with sentence {i+1}: ", round(sentence.similarity(keyword), 5))

>>> Similarity score with sentence 1:  0.26136
Similarity score with sentence 2:  0.14021
Similarity score with sentence 3:  0.13885

演習に進みましょう！

spaCyで学ぶNatural Language Processing