One-Hot Encoding
使用 PySpark 的機器學習
Andrew Collier
Data Scientist, Fathom Data
索引值的問題
# Counts for 'type' category
+-------+-----+
| type|count|
+-------+-----+
|Midsize| 22|
| Small| 21|
|Compact| 16|
| Sporty| 14|
| Large| 11|
| Van| 9|
+-------+-----+
# Numerical indices for 'type' category
+-------+--------+
| type|type_idx|
+-------+--------+
|Midsize| 0.0|
| Small| 1.0|
|Compact| 2.0|
| Sporty| 3.0|
| Large| 4.0|
| Van| 5.0|
+-------+--------+
虛擬變數（Dummy variables）
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | X | | | | | |
| Small| | | X | | | | |
|Compact| ===> | | | X | | | |
| Sporty| | | | | X | | |
| Large| | | | | | X | |
| Van| | | | | | | X |
+-------+ +-------+-------+-------+-------+-------+-------+
每個類別層級都成為一個欄位。
虛擬變數：二元編碼
+-------+ +-------+-------+-------+-------+-------+-------+
| type| |Midsize| Small|Compact| Sporty| Large| Van|
+-------+ +-------+-------+-------+-------+-------+-------+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 |
| Small| | 0 | 1 | 0 | 0 | 0 | 0 |
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 |
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 |
| Large| | 0 | 0 | 0 | 0 | 1 | 0 |
| Van| | 0 | 0 | 0 | 0 | 0 | 1 |
+-------+ +-------+-------+-------+-------+-------+-------+
二元值表示對應層級的有（1）或無（0）。
虛擬變數：稀疏表示法
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| Van| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | 0 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | 1 | | 5| 1|
+-------+ +-------+-------+-------+-------+-------+-------+ +------+-----+
稀疏表示法：只儲存欄索引與值。
虛擬變數：冗餘欄位
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
| type| |Midsize| Small|Compact| Sporty| Large| |Column|Value|
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
|Midsize| | 1 | 0 | 0 | 0 | 0 | | 0| 1|
| Small| | 0 | 1 | 0 | 0 | 0 | | 1| 1|
|Compact| ===> | 0 | 0 | 1 | 0 | 0 | ===> | 2| 1|
| Sporty| | 0 | 0 | 0 | 1 | 0 | | 3| 1|
| Large| | 0 | 0 | 0 | 0 | 1 | | 4| 1|
| Van| | 0 | 0 | 0 | 0 | 0 | | | |
+-------+ +-------+-------+-------+-------+-------+ +------+-----+
層級互斥，因此可刪掉一欄。
One-hot encoding
from pyspark.ml.feature import OneHotEncoder
onehot = OneHotEncoder(inputCols=['type_idx'], outputCols=['type_dummy'])
將編碼器套用到資料。
onehot = onehot.fit(cars)
# How many category levels?
onehot.categorySizes
[6]
One-hot encoding
cars = onehot.transform(cars)
cars.select('type', 'type_idx', 'type_dummy').distinct().sort('type_idx').show()
+-------+--------+-------------+
| type|type_idx| type_dummy|
+-------+--------+-------------+
|Midsize| 0.0|(5,[0],[1.0])|
| Small| 1.0|(5,[1],[1.0])|
|Compact| 2.0|(5,[2],[1.0])|
| Sporty| 3.0|(5,[3],[1.0])|
| Large| 4.0|(5,[4],[1.0])|
| Van| 5.0| (5,[],[])|
+-------+--------+-------------+
稠密與稀疏
from pyspark.mllib.linalg import DenseVector, SparseVector
儲存這個向量：[1, 0, 0, 0, 0, 7, 0, 0]。
DenseVector([1, 0, 0, 0, 0, 7, 0, 0])
DenseVector([1.0, 0.0, 0.0, 0.0, 0.0, 7.0, 0.0, 0.0])
SparseVector(8, [0, 5], [1, 7])
SparseVector(8, {0: 1.0, 5: 7.0})
為類別變數做 One-Hot 編碼
使用 PySpark 的機器學習
