以資料概化進行匿名化

Data Privacy and Anonymization in Python

Rebeca Gonzalez

Data engineer

資料概化

一種以較不精確的值取代資料值的技術。

目的是在保留分析實用性的同時，去除部分識別資訊。

可把資料值換成較不精確的類別：「Dancer」→「Artist」

     name               location
0    Amanda Hooper      146 Rodgers Field\nGregoryview, MS 71630
1    Sarah Smith        817 Garcia Shoal\nJonesville, AR 30299
2    Sean Boyd III      1938 90th St\nDallas, TX 59715

     name               location
0    Amanda Hooper      998 Boone Estate\nReedborough, MS 71630
1    Sarah Smith        7255 Shelby Rapids Apt. 455\nKarenland, AK 30299
2    Sean Boyd III      791 Crist Parks\nGreenton, TX 59715
Data Privacy and Anonymization in Python

資料概化

# Original data
df_employees.head()

     first name   last name    age   ssn
0    Amber        Brown        91    798-29-4785
1    William      Gibson       34    431-66-8381
2    Daniel       Lee          92    825-91-5550
3    Andrea       Stevenson    64    188-59-3544
4    Julie        Horn         35    020-60-6388

# Generalized data: Age in intervals and masked SSN
generalized_df.head()

    First name    Last name    Age        SSN
0    Amber        Brown        (80, 99]    798-**-****
1    William      Gibson       (30, 50]    431-**-****
2    Daniel       Lee          (80, 99]    825-**-****
3    Andrea       Stevenson    (60, 80]    188-**-****
4    Julie        Horn         (30, 50]    020-**-****

34 歲的人會被歸入 30 到 50 的區間。這也稱為分箱（binning）。

Data Privacy and Anonymization in Python

資料彙總

左側列出各種職業，右側對應較高層的一般類別。

Data Privacy and Anonymization in Python

醫療資料集

# Explore the dataset
df_medical.head()

    age    gender    department    condition
0    30    F         Finance       Anxiety disorders
1    42    M         Production    Bronchitis
2    35    F         Marketing     Dysthymia
3    39    F         Production    Dysthymia
4    40    M         Marketing     Flu
Data Privacy and Anonymization in Python

醫療資料集

# Explore the histogram of the age variable
df_medical['age'].hist(bins=15)

使用 hist 方法繪製的資料集中 age 變數的直方圖

Data Privacy and Anonymization in Python

概化

# Apply generalization by transforming to binary data
df_medical['age'] = df_medical['age'].apply(lambda x:">=40" if x>=40 else "<40" )


# See results
df_medical.head()

    age    gender    department    condition
0    <40   F         Finance       Bronchitis
1    >=40  M         Production    Bronquitis
2    <40   F         Finance       Dysthymia
3    <40   F         Production    Dysthymia
4    >=40  M         Marketing     Flu
Data Privacy and Anonymization in Python

頂端與底端編碼

# Explore the histogram of the age variable
df_medical['age'].hist(bins=15)

使用 hist 方法繪製的資料集中 age 變數的直方圖

  • 少數人小於 25 歲，也少數人高於 55 歲
  • 對離群者設上下界，以降低再識別風險
  • 類別內觀測很少時最適用，特別是在分配尾端
Data Privacy and Anonymization in Python

頂端編碼（Top coding）

# Filter to see rows affected
df_medical[df_medical['age'] >= 55]

      age   gender   department     condition
26    56    F        Production     Flu
65    55    M        Finance        Dysthymia
126   59    F        Production     Anxiety disorders
139   58    F        Finance        Dysthymia
142   59    M        Marketing      Flu
145   57    M        Marketing      Anxiety disorders
Data Privacy and Anonymization in Python

設定頂端編碼

# Top code the age to 55
df_medical.loc[df_medical['age'] > 55, 'age'] = 55


# Filter to see rows affected
df_medical[df_medical['age'] >= 55]

      age   gender   department     condition
26    55    F        Production     Flu
65    55    M        Finance        Dysthymia
126   55    F        Production     Anxiety disorders
139   55    F        Finance        Dysthymia
142   55    M        Marketing      Flu
145   55    M        Marketing      Anxiety disorders
Data Privacy and Anonymization in Python

底端編碼（Bottom coding）

# Explore the histogram of the age variable
df_medical['age'].hist(bins=15)

套用頂端編碼後，使用 hist 方法繪製的 age 直方圖；右側高值離群不再出現

Data Privacy and Anonymization in Python

設定底端編碼

# Bottom code the age to 25
df_medical.loc[df['age'] < 25, 'age'] = 25


# Explore the histogram of the age variable
df_medical['age'].hist(bins=15)

套用頂端與底端編碼後，使用 hist 方法繪製的 age 直方圖；已無離群值

Data Privacy and Anonymization in Python

資料概化與隱私模型

搭配抑制與遮罩，並遵循 隱私模型（如 K-anonymity）效果更佳。

指定資料集必須滿足的條件，以控制揭露風險。

下一章會學如何實作這個隱私模型！

Data Privacy and Anonymization in Python

一起來練習吧！

Data Privacy and Anonymization in Python

Preparing Video For Download...