加载数据

使用 PySpark 进行机器学习

Andrew Collier

Data Scientist, Fathom Data

DataFrame 快速回顾

用于表格数据的 DataFrame。

一个表示 Spark DataFrame 的数据表。

常用方法：

count()
show()
printSchema()

常用属性：

dtypes

汽车的 CSV 数据

"cars.csv" 文件的前几行。

mfr,mod,org,type,cyl,size,weight,len,rpm,cons
Mazda,RX-7,non-USA,Sporty,NA,1.3,2895,169,6500,9.41
Nissan,Maxima,non-USA,Midsize,6,3,3200,188,5200,9.05
Chevrolet,Cavalier,USA,Compact,4,2.2,2490,182,5200,6.53
Subaru,Legacy,non-USA,Compact,4,2.2,3085,179,5600,7.84
Ford,Escort,USA,Small,4,1.8,2530,171,6500,7.84

四辆黑色汽车。

从 CSV 读取数据

.csv() 方法读取 CSV 文件并返回 DataFrame。

cars = spark.read.csv('cars.csv', header=True)

可选参数：

header — 首行是否为表头？（默认：False）
sep — 字段分隔符（默认：逗号 ','）
schema — 显式列数据类型
inferSchema — 从数据推断列类型
nullValue — 缺失值占位符

快速查看数据

DataFrame 的前五条记录。

cars.show(5)

+---------+--------+-------+-------+---+----+------+---+----+----+
|      mfr|     mod|    org|   type|cyl|size|weight|len| rpm|cons|
+---------+--------+-------+-------+---+----+------+---+----+----+
|    Mazda|    RX-7|non-USA| Sporty| NA| 1.3|  2895|169|6500|9.41|
|   Nissan|  Maxima|non-USA|Midsize|  6|   3|  3200|188|5200|9.05|
|Chevrolet|Cavalier|    USA|Compact|  4| 2.2|  2490|182|5200|6.53|
|   Subaru|  Legacy|non-USA|Compact|  4| 2.2|  3085|179|5600|7.84|
|     Ford|  Escort|    USA|  Small|  4| 1.8|  2530|171|6500|7.84|
+---------+--------+-------+-------+---+----+------+---+----+----+

检查列类型

cars.printSchema()

 root
 |-- mfr: string (nullable = true)
 |-- mod: string (nullable = true)
 |-- org: string (nullable = true)
 |-- type: string (nullable = true)
 |-- cyl: string (nullable = true)
 |-- size: string (nullable = true)
 |-- weight: string (nullable = true)
 |-- len: string (nullable = true)
 |-- rpm: string (nullable = true)
 |-- cons: string (nullable = true)

从数据推断列类型

cars = spark.read.csv("cars.csv", header=True, inferSchema=True)
cars.dtypes

 [('mfr', 'string'),
 ('mod', 'string'),
 ('org', 'string'),
 ('type', 'string'),
 ('cyl', 'string'),
 ('size', 'double'),
 ('weight', 'int'),
 ('len', 'int'),
 ('rpm', 'int'),
 ('cons', 'double')]

处理缺失数据

使用 nullValue 参数处理缺失数据。

cars = spark.read.csv("cars.csv", header=True, inferSchema=True, nullValue='NA')

nullValue 区分大小写。

指定列类型

schema = StructType([
    StructField("maker", StringType()),
    StructField("model", StringType()),
    StructField("origin", StringType()),
    StructField("type", StringType()),
    StructField("cyl", IntegerType()),
    StructField("size", DoubleType()),
    StructField("weight", IntegerType()),
    StructField("length", DoubleType()),
    StructField("rpm", IntegerType()),
    StructField("consumption", DoubleType())
])
cars = spark.read.csv("cars.csv", header=True, schema=schema, nullValue='NA')

最终的汽车数据

+----------+-------------+-------+-------+----+----+------+------+----+-----------+
|maker     |model        |origin |type   |cyl |size|weight|length|rpm |consumption|
+----------+-------------+-------+-------+----+----+------+------+----+-----------+
|Mazda     |RX-7         |non-USA|Sporty |null|1.3 |2895  |169.0 |6500|9.41       |
|Nissan    |Maxima       |non-USA|Midsize|6   |3.0 |3200  |188.0 |5200|9.05       |
|Chevrolet |Cavalier     |USA    |Compact|4   |2.2 |2490  |182.0 |5200|6.53       |
|Subaru    |Legacy       |non-USA|Compact|4   |2.2 |3085  |179.0 |5600|7.84       |
|Ford      |Escort       |USA    |Small  |4   |1.8 |2530  |171.0 |6500|7.84       |
|Mercury   |Capri        |USA    |Sporty |4   |1.6 |2450  |166.0 |5750|9.05       |
|Oldsmobile|Cutlass Ciera|USA    |Midsize|4   |2.2 |2890  |190.0 |5200|7.59       |
|Saab      |900          |non-USA|Compact|4   |2.1 |2775  |184.0 |6000|9.05       |
|Dodge     |Caravan      |USA    |Van    |6   |3.0 |3705  |175.0 |5000|11.2       |
+----------+-------------+-------+-------+----+----+------+------+----+-----------+

让我们加载一些数据！

使用 PySpark 进行机器学习