spaCy के साथ semantic similarity मापना

spaCy के साथ Natural Language Processing

Azadeh Mobasher

Principal Data Scientist

The semantic similarity method

टेक्स्ट का विश्लेषण करके समानताएँ पहचानने की प्रक्रिया
टेक्स्ट को पहले से तय श्रेणियों में बाँटना या प्रासंगिक टेक्स्ट खोजना
Similarity score बताता है कि दो टेक्स्ट कितने समान हैं

What is the cheapest flight from Boston to Seattle?
Which airline serves Denver, Pittsburgh and Atlanta?
What kinds of planes are used by American Airlines?

Similarity score

टेक्स्ट पर परिभाषित एक मेट्रिक
समानता मापने के लिए Cosine similarity और word vectors का उपयोग करें
Cosine similarity 0 और 1 के बीच कोई भी संख्या होती है

Cosine similarity and vectors

Token similarity

spaCy Token objects के बीच similarity scores निकालता है

nlp = spacy.load("en_core_web_md")
doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

token1 = doc1[2]
token2 = doc2[4]
print(f"Similarity between {token1} and {token2} = ", round(token1.similarity(token2), 3))

>>> Similarity between pizza and pasta =  0.685

Span similarity

spaCy दिए गए दो Span objects की semantic similarity निकालता है

doc1 = nlp("We eat pizza")
doc2 = nlp("We like to eat pasta")

span1 = doc1[1:]
span2 = doc2[1:]

print(f"Similarity between \"{span1}\" and \"{span2}\" = ",
        round(span1.similarity(span2), 3))

>>> Similarity between "eat pizza" and "like to eat pasta" =  0.588

print(f"Similarity between \"{doc1[1:]}\" and \"{doc2[3:]}\" = ",
        round(doc1[1:].similarity(doc2[3:]), 3))

>>> Similarity between "eat pizza" and "eat pasta" =  0.936

Doc similarity

spaCy दो डॉक्यूमेंट्स के बीच similarity scores निकालता है

nlp = spacy.load("en_core_web_md")

doc1 = nlp("I like to play basketball")
doc2 = nlp("I love to play basketball")
print("Similarity score :", round(doc1.similarity(doc2), 3))

>>> Similarity score : 0.975

उच्च cosine similarity बहुत अधिक semantic समानता दिखाती है
Doc vectors डिफ़ॉल्ट रूप से word vectors का average होते हैं

Sentence similarity

spaCy किसी दिए गए कीवर्ड से संबंधित content ढूँढता है
शब्द price से मिलते-जुलते ग्राहक प्रश्न खोजना:

sentences = nlp("What is the cheapest flight from Boston to Seattle? 
                 Which airline serves Denver, Pittsburgh and Atlanta? 
                 What kinds of planes are used by American Airlines?")

keyword = nlp("price")

for i, sentence in enumerate(sentences.sents):
    print(f"Similarity score with sentence {i+1}: ", round(sentence.similarity(keyword), 5))

>>> Similarity score with sentence 1:  0.26136
Similarity score with sentence 2:  0.14021
Similarity score with sentence 3:  0.13885

अभ्यास करते हैं!

spaCy के साथ Natural Language Processing