Hugging Face Datasets

善用 Hugging Face

Jacob H. Marquez

Lead Data Engineer

Hugging Face Datasets 圖示

¹ https://huggingface.co/datasets

Hugging Face Datasets 介面

¹ https://huggingface.co/datasets

安裝 Datasets 套件

pip install datasets

🌐 存取
📥 下載
🔧 使用
🤝 分享

Hugging Face Datasets 標誌

¹ https://huggingface.co/docs/datasets/loading

下載資料集

from datasets import load_dataset

data = load_dataset("IVN-RIN/BioBERT_Italian")

Split 參數

data = load_dataset("IVN-RIN/BioBERT_Italian", split="train")

¹ https://huggingface.co/docs/datasets/v2.15.0/loading

Apache Arrow 資料集格式

Apache Arrow 資料集

¹ https://arrow.apache.org/overview/

資料操作

data = load_dataset("IVN-RIN/BioBERT_Italian", split="train")


# 篩選出包含模式 " bella "
filtered = data.filter(lambda row: " bella " in row['text'])
print(filtered)

Dataset({
    features: ['text'],
    num_rows: 1122
})

¹ https://huggingface.co/docs/datasets/process#select-and-filter

資料操作

# 取前兩列
sliced = filtered.select(range(2))


print(sliced)

Dataset({features: ['text'], num_rows: 2})

# 取第 1 列的 'text'
print(sliced[0]['text'])

Concentrazioni atmosferiche di PCDD/PCDF...

¹ https://huggingface.co/docs/datasets/process#select-and-filter

一起來練習吧！

善用 Hugging Face