Tokenization और Lemmatization

Python में NLP के लिए Feature Engineering

Rounak Banik

Data Scientist

Text sources

News articles
Tweets
Comments

टेक्स्ट को मशीन-फ्रेंडली बनाना

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

टेक्स्ट प्रीप्रोसेसिंग तकनीकें

शब्दों को lowercase में बदलना
शुरुआती और अंतिम whitespaces हटाना
punctuation हटाना
stopwords हटाना
contractions फैलाना
विशेष कैरेक्टर हटाना (numbers, emojis, आदि)

Tokenization

"I have a dog. His name is Hachi."

Tokens:

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

Tokens:

["Do", "n't", "do", "this", "."]

spaCy से Tokenization

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

Lemmatization

शब्द को उसके base form में बदलें
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

spaCy से Lemmatization

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

अभ्यास करते हैं!

Python में NLP के लिए Feature Engineering