Additional operators

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

PythonOperator

Executes a Python function / callable
Operates similarly to the BashOperator, with more options
Can pass in arguments to the Python code

from airflow.operators.python import PythonOperator

def printme():
    print("This goes in the logs!")

python_task = PythonOperator(
    task_id='simple_print',
    python_callable=printme
)

Arguments

Supports arguments to tasks
- Positional
- Keyword
Use the op_kwargs dictionary

op_kwargs example

def sleep(length_of_time):
  time.sleep(length_of_time)


sleep_task = PythonOperator(
    task_id='sleep',
    python_callable=sleep,
    op_kwargs={'length_of_time': 5}
)

EmailOperator

Found in the airflow.operators library
Sends an email
Can contain typical components
- HTML content
- Attachments
Does require the Airflow system to be configured with email server details

EmailOperator example

from airflow.operators.email import EmailOperator


email_task = EmailOperator(
    task_id='email_sales_report',
    to='[email protected]',
    subject='Automated Sales Report',
    html_content='Attached is the latest sales report',
    files='latest_sales.xlsx'
)

Let's practice!

Introduction to Apache Airflow in Python