One-Hot Encoding
Machine Learning con PySpark
Andrew Collier
Data Scientist, Fathom Data
Il problema dei valori indicizzati
# Conteggi per la categoria 'type'
+-------+-----+
| type|count|
+-------+-----+
|Midsize| 22|
| Small| 21|
|Compact| 16|
| Sporty| 14|
| Large| 11|
| Van| 9|
+-------+-----+
# Indici numerici per la categoria 'type'
+-------+--------+
| type|type_idx|
+-------+--------+
|Midsize| 0.0|
| Small| 1.0|
|Compact| 2.0|
| Sporty| 3.0|
| Large| 4.0|
| Van| 5.0|
+-------+--------+
Variabili dummy
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | X | | | | | |
| Small| | | X | | | | |
|Compact| ===> | | | X | | | |
| Sporty| | | | | X | | |
| Large| | | | | | X | |
| Van| | | | | | | X |
+-------+ +-------+-------+-------+-------+-------+-------+
Ogni livello categoriale diventa una colonna.
Variabili dummy: codifica binaria
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 |
| Small| | 0 | 1 | 0 | 0 | 0 | 0 |
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 |
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 |
| Large| | 0 | 0 | 0 | 0 | 1 | 0 |
| Van| | 0 | 0 | 0 | 0 | 0 | 1 |
+-------+ +-------+-------+-------+-------+-------+-------+
Valori binari indicano presenza (1) o assenza (0) del livello corrispondente.
Variabili dummy: rappresentazione sparsa
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| Van| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | 0 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | 1 | | 5| 1|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
Rappresentazione sparsa: salva indice colonna e valore.
Variabili dummy: colonna ridondante
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | | | |
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
I livelli sono mutualmente esclusivi: eliminale uno.
One-hot encoding
from pyspark.ml.feature import OneHotEncoder
onehot = OneHotEncoder(inputCols=['type_idx'], outputCols=['type_dummy'])
Adatta l'encoder ai dati.
onehot = onehot.fit(cars)
# Quanti livelli di categoria?
onehot.categorySizes
[6]
One-hot encoding
cars = onehot.transform(cars)
cars.select('type', 'type_idx', 'type_dummy').distinct().sort('type_idx').show()
+-------+--------+-------------+
| type|type_idx| type_dummy|
+-------+--------+-------------+
|Midsize| 0.0|(5,[0],[1.0])|
| Small| 1.0|(5,[1],[1.0])|
|Compact| 2.0|(5,[2],[1.0])|
| Sporty| 3.0|(5,[3],[1.0])|
| Large| 4.0|(5,[4],[1.0])|
| Van| 5.0| (5,[],[])|
+-------+--------+-------------+
Denso vs sparso
from pyspark.mllib.linalg import DenseVector, SparseVector
Memorizza questo vettore: [1, 0, 0, 0, 0, 7, 0, 0].
DenseVector([1, 0, 0, 0, 0, 7, 0, 0])
DenseVector([1.0, 0.0, 0.0, 0.0, 0.0, 7.0, 0.0, 0.0])
SparseVector(8, [0, 5], [1, 7])
SparseVector(8, {0: 1.0, 5: 7.0})
One-Hot: variabili categoriche
Machine Learning con PySpark
