Tokenisierung und Lemmatisierung

Feature Engineering für NLP in Python

Rounak Banik

Data Scientist

Textquellen

Nachrichtenartikel
Tweets
Kommentare

Text maschinenfreundlich machen

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

Techniken der Textvorverarbeitung

Wörter in Kleinbuchstaben umwandeln
Leerzeichen am Anfang/Ende entfernen
Satzzeichen entfernen
Stoppwörter entfernen
Kontraktionen ausschreiben
Sonderzeichen entfernen (Zahlen, Emojis etc.)

Tokenisierung

"I have a dog. His name is Hachi."

Tokens:

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

Tokens:

["Do", "n't", "do", "this", "."]

Tokenisierung mit spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

Lemmatisierung

Wort auf Grundform abbilden
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

Lemmatisierung mit spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

Lass uns üben!

Feature Engineering für NLP in Python