使用迭代器将大文件载入内存

Python 工具箱

Hugo Bowne-Anderson

Data Scientist at DataCamp

分块加载数据

数据量可能过大，无法全部放入内存
解决方案：分块读取数据！
pandas 函数：read_csv()
- 指定分块：chunksize

迭代数据

import pandas as pd
result = []

for chunk in pd.read_csv('data.csv', chunksize=1000):

    result.append(sum(chunk['x']))

total = sum(result)

print(total)

迭代数据

import pandas as pd
total = 0

for chunk in pd.read_csv('data.csv', chunksize=1000):
    total += sum(chunk['x'])

print(total)

开始练习！

Python 工具箱