Introduzione al feature engineering per l'NLP

Feature Engineering per NLP in Python

Rounak Banik

Data Scientist

Dati numerici

Dataset Iris

sepal length sepal width petal length petal width class
6.3 2.9 5.6 1.8 Iris-virginica
4.9 3.0 1.4 0.2 Iris-setosa
5.6 2.9 3.6 1.3 Iris-versicolor
6.0 2.7 5.1 1.6 Iris-versicolor
7.2 3.6 6.1 2.5 Iris-virginica
Feature Engineering per NLP in Python

One-hot encoding

sex
female
male
female
male
female
...
Feature Engineering per NLP in Python

One-hot encoding

sex one-hot encoding
female →
male →
female →
male →
female →
... ...
Feature Engineering per NLP in Python

One-hot encoding

sex one-hot encoding sex_female sex_male
female → 1 0
male → 0 1
female → 1 0
male → 0 1
female → 1 0
... ... ... ...
Feature Engineering per NLP in Python

One-hot encoding con pandas

# Import the pandas library
import pandas as pd


# Perform one-hot encoding on the 'sex' feature of df
df = pd.get_dummies(df, columns=['sex'])
Feature Engineering per NLP in Python

Dati testuali

Movie Review Dataset

review class
This movie is for dog lovers. A very poignant... positive
The movie is forgettable. The plot lacked... negative
A truly amazing movie about dogs. A gripping... positive
Feature Engineering per NLP in Python

Pre-processing del testo

  • Conversione in minuscolo
    • Esempio: Reduction → reduction
  • Conversione alla forma base
    • Esempio: reduction → reduce
Feature Engineering per NLP in Python

Vettorizzazione

review class
This movie is for dog lovers. A very poignant... positive
The movie is forgettable. The plot lacked... negative
A truly amazing movie about dogs. A gripping... positive
Feature Engineering per NLP in Python

Vettorizzazione

0 1 2 ... n class
0.03 0.71 0.00 ... 0.22 positive
0.45 0.00 0.03 ... 0.19 negative
0.14 0.18 0.00 ... 0.45 positive
Feature Engineering per NLP in Python

Feature di base

  • Numero di parole
  • Numero di caratteri
  • Lunghezza media delle parole
  • Tweet

Tweet di Silverado Records

Feature Engineering per NLP in Python

POS tagging

Word POS
I Pronoun
have Verb
a Article
dog Noun
Feature Engineering per NLP in Python

Named Entity Recognition

  • Il sostantivo si riferisce a persona, organizzazione o paese?

Una persona, la bandiera di un paese e il logo di TED

Noun NER
Brian Person
DataCamp Organization
Feature Engineering per NLP in Python

Concetti trattati

  • Pre-processing del testo
  • Feature di base
  • Feature sulle parole
  • Vettorizzazione
Feature Engineering per NLP in Python

Esercitiamoci!

Feature Engineering per NLP in Python

Preparing Video For Download...