將非結構化資料轉為 DataFrame

在 Python 中使用 Dask 進行平行程式設計

James Fulton

Climate Informatics Researcher

巢狀 JSON 資料

在範例 JSON 檔 example_0.json 中

{"name": "Beth", "employment": [{"role": "manager", "start_date": ...}, ...], ...}
{"name": "Omar", "employment": [{"role": "analyst", "start_date": ...}, ...], ...}
{"name": "Fang", "employment": [{"role": "engineer", "start_date": ...}, ...], ...}
...

重組字典結構

def add_number_of_jobs(employee_dict):
    employee_dict['number_of_previous_jobs'] = len(employee_dict['employment'])
    return employee_dict

dict_bag = dict_bag.map(add_number_of_jobs)

移除字典部分內容

def delete_dictionary_entry(dictionary, key_to_drop):
    del dictionary[key_to_drop]
    return dictionary

dict_bag = dict_bag.map(delete_dictionary_entry, key_to_drop='employment')

選取字典部分欄位

def filter_dictionary(dictionary, keys_to_keep):
    new_dict = {}
    for k in keys_to_keep:
        new_dict[k] = dictionary[k]
    return new_dict

dict_bag = dict_bag.map(
    filter_dictionary, 
    keys_to_keep=['name', 'number_of_previous_jobs']
)

轉換為 DataFrame

print(dict_bag.take(1))

({'name': 'Beth',
  'number_of_previous_jobs': 3},)

converted_bag_df = dict_bag.to_dataframe()

print(converted_bag_df)

                 name    number_of_previous_jobs    
npartitions=3
               object                    float64
                  ...                        ...

一起來練習吧！

在 Python 中使用 Dask 進行平行程式設計