Semantic search और enriched embeddings

OpenAI API के साथ Embeddings परिचय

Emmanuel Pire

Senior Software Engineer, DataCamp

Semantic search

Embeddings का उपयोग करके सर्च क्वेरी के सबसे मिलते-जुलते रिजल्ट लौटाएँ
उदाहरण: ऑनलाइन न्यूज़ वेबसाइट के लिए semantic search

Semantic search कैसे काम करता है: embedding मॉडल सर्च टेक्स्ट को embed करता है, फिर embedded सर्च टेक्स्ट और embedded हेडलाइन्स के बीच दूरी मापी जाती है. सबसे नज़दीकी हेडलाइन लौटाई जाती है.

Semantic search

सर्च क्वेरी और अन्य टेक्स्ट को embed करें
cosine distances निकालें
सबसे छोटी cosine distance वाले टेक्स्ट को extract करें

Enriched embeddings

articles = [
    {"headline": "Economic Growth Continues Amid Global Uncertainty",
     "topic": "Business",
     "keywords": ["economy", "business", "finance"]},
    ...
    {"headline": "1.5 Billion Tune-in to the World Cup Final",
     "topic": "Sport",
     "keywords": ["soccer", "world cup", "tv"]}
]

Headline: Economic Growth Continues Amid Global Uncertainty
Topic: Business
Keywords: economy, business, finance

F-strings से फीचर्स जोड़ना

articles = [..., {"headline": "1.5 Billion Tune-in to the World Cup ",
                  "topic": "Sport",
                  "keywords": ["soccer", "world cup", "tv"]}]


def create_article_text(article):

  return f"""Headline: {article['headline']}
Topic: {article['topic']}
Keywords: {', '.join(article['keywords'])}"""


print(create_article_text(articles[-1]))

Headline: 1.5 Billion Tune-in to the World Cup Final
Topic: Sport
Keywords: soccer, world cup, tv

Enriched embeddings बनाना

article_texts = [create_article_text(article) for article in articles]


article_embeddings = create_embeddings(article_texts)

print(article_embeddings)

[[-0.019609929993748665, -0.03331860154867172, ...],
 ...,
 [..., -0.014373429119586945, -0.005235843360424042]]

Distances की गणना

from scipy.spatial import distance

def find_n_closest(query_vector, embeddings, n=3):

  distances = []
  for index, embedding in enumerate(embeddings):
    dist = distance.cosine(query_vector, embedding)
    distances.append({"distance": dist, "index": index})

  distances_sorted = sorted(distances, key=lambda x: x["distance"])

  return distances_sorted[0:n]

सर्च परिणाम लौटाना

query_text = "AI"

query_vector = create_embeddings(query_text)[0]


hits = find_n_closest(query_vector, article_embeddings)


for hit in hits:
  article = articles[hit['index']]
  print(article['headline'])

Tech Giant Buys 49% Stake In AI Startup
Tech Company Launches Innovative Product to Improve Online Accessibility
India Successfully Lands Near Moon's South Pole

अभ्यास करते हैं!

OpenAI API के साथ Embeddings परिचय