Tokenisering och lemmatisering

Funktionsutveckling för NLP i Python

Rounak Banik

Data Scientist

Textkällor

Nyhetsartiklar
Tweets
Kommentarer

Göra text maskinvänlig

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

Tekniker för textförbehandling

Konvertera ord till gemener
Ta bort inledande och avslutande blanksteg
Ta bort interpunktion
Ta bort stoppord
Expandera sammandragningar
Ta bort specialtecken (siffror, emojis, osv.)

Tokenisering

"I have a dog. His name is Hachi."

Tokens:

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

Tokens:

["Do", "n't", "do", "this", "."]

Tokenisering med spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

Lemmatisering

Konvertera ord till grundform
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

Lemmatisering med spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

Nu kör vi en övning!

Funktionsutveckling för NLP i Python