Kodowanie One-Hot
Uczenie maszynowe z PySpark
Andrew Collier
Data Scientist, Fathom Data
Problem z wartościami indeksowanymi
# Counts for 'type' category
+-------+-----+
| type|count|
+-------+-----+
|Midsize| 22|
| Small| 21|
|Compact| 16|
| Sporty| 14|
| Large| 11|
| Van| 9|
+-------+-----+
# Numerical indices for 'type' category
+-------+--------+
| type|type_idx|
+-------+--------+
|Midsize| 0.0|
| Small| 1.0|
|Compact| 2.0|
| Sporty| 3.0|
| Large| 4.0|
| Van| 5.0|
+-------+--------+
Zmienne dummy
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | X | | | | | |
| Small| | | X | | | | |
|Compact| ===> | | | X | | | |
| Sporty| | | | | X | | |
| Large| | | | | | X | |
| Van| | | | | | | X |
+-------+ +-------+-------+-------+-------+-------+-------+
Każdy poziom kategoryczny staje się kolumną.
Zmienne dummy: kodowanie binarne
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 |
| Small| | 0 | 1 | 0 | 0 | 0 | 0 |
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 |
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 |
| Large| | 0 | 0 | 0 | 0 | 1 | 0 |
| Van| | 0 | 0 | 0 | 0 | 0 | 1 |
+-------+ +-------+-------+-------+-------+-------+-------+
Wartości binarne wskazują obecność (1) lub brak (0) danego poziomu.
Zmienne dummy: reprezentacja rzadka
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| Van| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | 0 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | 1 | | 5| 1|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
Reprezentacja rzadka: przechowuje indeks kolumny i wartość.
Zmienne dummy: nadmiarowa kolumna
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | | | |
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
Poziomy są rozłączne, więc jeden należy usunąć.
Kodowanie one-hot
from pyspark.ml.feature import OneHotEncoder
onehot = OneHotEncoder(inputCols=['type_idx'], outputCols=['type_dummy'])
Dopasowanie kodera do danych.
onehot = onehot.fit(cars)
# How many category levels?
onehot.categorySizes
[6]
Kodowanie one-hot
cars = onehot.transform(cars)
cars.select('type', 'type_idx', 'type_dummy').distinct().sort('type_idx').show()
+-------+--------+-------------+
| type|type_idx| type_dummy|
+-------+--------+-------------+
|Midsize| 0.0|(5,[0],[1.0])|
| Small| 1.0|(5,[1],[1.0])|
|Compact| 2.0|(5,[2],[1.0])|
| Sporty| 3.0|(5,[3],[1.0])|
| Large| 4.0|(5,[4],[1.0])|
| Van| 5.0| (5,[],[])|
+-------+--------+-------------+
Gęsty a rzadki
from pyspark.mllib.linalg import DenseVector, SparseVector
Zapis wektora: [1, 0, 0, 0, 0, 7, 0, 0].
DenseVector([1, 0, 0, 0, 0, 7, 0, 0])
DenseVector([1.0, 0.0, 0.0, 0.0, 0.0, 7.0, 0.0, 0.0])
SparseVector(8, [0, 5], [1, 7])
SparseVector(8, {0: 1.0, 5: 7.0})
Kodowanie one-hot zmiennych kategorycznych
Uczenie maszynowe z PySpark
