User defined functions

Cleaning Data with PySpark

Mike Metzger

Data Engineering Consultant

Defined...

User defined functions or UDFs

Python method
Wrapped via the pyspark.sql.functions.udf method
Stored as a variable
Called like a normal Spark function

Reverse string UDF

Define a Python method

def reverseString(mystr):
    return mystr[::-1]

Wrap the function and store as a variable

udfReverseString = udf(reverseString, StringType())

Use with Spark

user_df = user_df.withColumn('ReverseName', 
                 udfReverseString(user_df.Name))

Argument-less example

def sortingCap():
    return random.choice(['G', 'H', 'R', 'S'])
udfSortingCap = udf(sortingCap, StringType())
user_df = user_df.withColumn('Class', udfSortingCap())

Name	Age	Class
Alice	14	H
Bob	18	S
Candice	63	G

Let's practice!

Cleaning Data with PySpark