分词与词形还原

Python 中的 NLP 特征工程

Rounak Banik

Data Scientist

文本来源

新闻报道
推文
评论

让文本更易于机器处理

Dogs, dog
reduction, REDUCING, Reduce
don't, do not
won't, will not

文本预处理技术

转为小写
移除首尾空白
移除标点
移除停用词
展开缩写
移除特殊字符（数字、表情等）

分词

"I have a dog. His name is Hachi."

词元（Tokens）：

["I", "have", "a", "dog", ".", "His", "name", "is", "Hachi", "."]

"Don't do this."

词元（Tokens）：

["Do", "n't", "do", "this", "."]

使用 spaCy 进行分词

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')

# Initiliaze string
string = "Hello! I don't know what I'm doing here."

# Create a Doc object
doc = nlp(string)

# Generate list of tokens
tokens = [token.text for token in doc]
print(tokens)

['Hello','!','I','do',"n't",'know','what','I',"'m",'doing','here','.']

词形还原

将词转为其基本形式
- reducing, reduces, reduced, reduction → reduce
- am, are, is → be
- n't → not
- 've → have

使用 spaCy 进行词形还原

import spacy

# Load the en_core_web_sm model
nlp = spacy.load('en_core_web_sm')
# Initiliaze string
string = "Hello! I don't know what I'm doing here."
# Create a Doc object
doc = nlp(string)


# Generate list of lemmas
lemmas = [token.lemma_ for token in doc]
print(lemmas)

['hello','!','-PRON-','do','not','know','what','-PRON','be','do','here', '.']

Vamos praticar!

Python 中的 NLP 特征工程