Tokenization en lemmatization

Feature Engineering voor NLP in Python

Rounak Banik

Data Scientist

Tekstbronnen

Nieuwsartikelen
Tweets
Reacties

Tekst machinevriendelijk maken

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

Tekstvoorbewerkingstechnieken

Woorden naar lowercase omzetten
Voor- en eindspaties verwijderen
Interpunctie verwijderen
Stopwoorden verwijderen
Samentrekkingen uitschrijven
Speciale tekens verwijderen (cijfers, emoji’s, enz.)

Tokenization

"I have a dog. His name is Hachi."

Tokens:

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

Tokens:

["Do", "n't", "do", "this", "."]

Tokenization met spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

Lemmatization

Zet woorden om naar de basisvorm
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

Lemmatization met spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

Laten we oefenen!

Feature Engineering voor NLP in Python