使用 spaCy 度量语义相似度

使用 spaCy 的自然语言处理

Azadeh Mobasher

Principal Data Scientist

语义相似度方法

分析文本以识别相似性的过程
将文本归入预设类别或检测相关文本
相似度分数衡量两段文本的相似程度

What is the cheapest flight from Boston to Seattle?
Which airline serves Denver, Pittsburgh and Atlanta?
What kinds of planes are used by American Airlines?

相似度分数

定义在文本上的一种度量
计算相似度使用余弦相似度和词向量
余弦相似度介于 0 到 1 之间

余弦相似度与向量

Token 相似度

spaCy 计算 Token 对象之间的相似度

nlp = spacy.load("en_core_web_md")
doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

token1 = doc1[2]
token2 = doc2[4]
print(f"Similarity between {token1} and {token2} = ", round(token1.similarity(token2), 3))

>>> Similarity between pizza and pasta =  0.685

Span 相似度

spaCy 计算两个 Span 对象的语义相似度

doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

span1 = doc1[1:]
span2 = doc2[1:]

print(f"Similarity between \"{span1}\" and \"{span2}\" = ",
        round(span1.similarity(span2), 3))

>>> Similarity between "eat pizza" and "like to eat pasta" =  0.588

print(f"Similarity between \"{doc1[1:]}\" and \"{doc2[3:]}\" = ",
        round(doc1[1:].similarity(doc2[3:]), 3))

>>> Similarity between "eat pizza" and "eat pasta" =  0.936

Doc 相似度

spaCy 计算两个文档之间的相似度

nlp = spacy.load("en_core_web_md")

doc1 = nlp("I like to play basketball")
doc2 = nlp("I love to play basketball")
print("Similarity score :", round(doc1.similarity(doc2), 3))

>>> Similarity score : 0.975

高余弦相似度表示语义高度相近
Doc 向量默认为词向量的平均值

句子相似度

spaCy 可根据关键词查找相关内容
查找与关键词 price 相似的客户问题：

sentences = nlp("What is the cheapest flight from Boston to Seattle? 
                 Which airline serves Denver, Pittsburgh and Atlanta? 
                 What kinds of planes are used by American Airlines?")

keyword = nlp("price")

for i, sentence in enumerate(sentences.sents):
    print(f"Similarity score with sentence {i+1}: ", round(sentence.similarity(keyword), 5))

>>> Similarity score with sentence 1:  0.26136
Similarity score with sentence 2:  0.14021
Similarity score with sentence 3:  0.13885

Passons à la pratique !

使用 spaCy 的自然语言处理