Tokenisasi dan Lemmatisasi

Rekayasa Fitur untuk NLP di Python

Rounak Banik

Data Scientist

Sumber teks

Artikel berita
Tweet
Komentar

Membuat teks ramah mesin

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

Teknik prapemrosesan teks

Ubah huruf ke lowercase
Hapus spasi di awal/akhir
Hapus tanda baca
Hapus stopword
Perluas kontraksi
Hapus karakter khusus (angka, emoji, dll.)

Tokenisasi

"I have a dog. His name is Hachi."

Token:

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

Token:

["Do", "n't", "do", "this", "."]

Tokenisasi dengan spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

Lemmatisasi

Ubah kata ke bentuk dasar
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

Lemmatisasi dengan spaCy

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

Ayo berlatih!

Rekayasa Fitur untuk NLP di Python