An Overview of Apache Airflow

Data Engineering
Workflow Automation
Apache Airflow
Author

Ravi Kalia

Published

March 31, 2025

Introduction

Apache Airflow is an open-source platform for orchestrating complex workflows. It allows you to define, schedule, and monitor workflows as Directed Acyclic Graphs (DAGs). It is widely used for data engineering, machine learning pipelines, and ETL processes due to its scalability, flexibility, and monitoring capabilities.

Why Use Apache Airflow?

Airflow is useful when you need: - Task scheduling: Automate workflow execution. - Dependency management: Ensure tasks run in a specific order. - Scalability: Run workflows on a single machine or a distributed cluster. - Monitoring & Logging: Track execution with a built-in web UI. - Integration: Connect with databases, cloud services, APIs, and more.

Why Use Airflow Instead of a Plain Python Script?

While a plain Python script can handle simple workflows, Airflow provides many advantages for complex workflows:

1. Built-in Scheduling and Dependency Management

  • Python scripts typically require manual scheduling (e.g., using cron or sleep loops), whereas Airflow allows you to set schedules declaratively.
  • Airflow ensures dependencies between tasks are correctly followed using DAGs, making workflows easier to maintain and debug.

2. Scalability

  • A Python script runs tasks sequentially or with manual threading, whereas Airflow can distribute tasks across multiple workers using CeleryExecutor or KubernetesExecutor.

3. Robust Monitoring & Logging

  • Airflow provides a web UI to visualize DAG execution, task retries, and logs, making debugging easier.
  • Python scripts require manual logging setup and tracking.

4. Fault Tolerance & Retries

  • Airflow allows defining automatic retries with exponential backoff, handling transient failures gracefully.
  • A Python script needs explicit error handling and retry logic.

5. Integration with External Systems

  • Airflow provides built-in operators and hooks for databases, cloud storage (AWS, GCP, Azure), APIs, and messaging systems (Kafka, RabbitMQ).
  • A Python script requires writing custom integration logic.

6. Dynamic Workflow Execution

  • Airflow DAGs can generate tasks dynamically based on conditions or external inputs.
  • Python scripts require explicit conditional logic and looping.

7. Versioning & Reproducibility

  • Airflow enables version-controlled workflows via Git, ensuring reproducibility.
  • Python scripts must be manually versioned.

If your workflow is simple and requires only a few steps, a Python script may suffice. But for scalable, maintainable, and robust workflows, Airflow is the better choice.

Core Concepts in Airflow

1. DAG (Directed Acyclic Graph)

A DAG represents a workflow with dependencies between tasks. Each DAG has a schedule, start date, and a set of tasks.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def my_task():
    print("Hello, Airflow!")

with DAG(dag_id="my_dag",
         start_date=datetime(2024, 3, 31),
         schedule_interval="@daily",
         catchup=False) as dag:
    
    task = PythonOperator(
        task_id="print_message",
        python_callable=my_task,
    )

Important DAG Methods

  • dag.run(): Manually trigger the DAG execution.
  • dag.get_task(task_id): Retrieve a specific task by ID.
  • dag.clear(): Reset task states so they can be re-run.

2. Operators

Operators define the type of tasks in a DAG. Some common operators include: - PythonOperator: Runs a Python function. - BashOperator: Executes shell commands. - EmailOperator: Sends email notifications. - SQLExecuteQueryOperator: Runs SQL queries.

Important Operator Methods

  • task.execute(context): Manually executes a task.
  • task.render_template_fields(context): Renders Jinja templates in the task.
  • task.set_upstream(task): Defines that a task must run before the given task.
  • task.set_downstream(task): Defines that a task must run after the given task.

3. Executors

Executors determine how tasks are run: - LocalExecutor: Runs tasks sequentially on a single machine. - CeleryExecutor: Distributes tasks across multiple workers. - KubernetesExecutor: Runs tasks in a Kubernetes cluster.

Important Executor Methods

  • executor.heartbeat(): Checks the status of running tasks.
  • executor.start(): Starts the execution engine.
  • executor.end(): Stops the execution engine.

4. Hooks and Sensors

  • Hooks connect to external systems (e.g., databases, cloud storage).
  • Sensors wait for an event before proceeding (e.g., file arrival, API response).
from airflow.sensors.filesystem import FileSensor

file_sensor = FileSensor(
    task_id="wait_for_file",
    filepath="/data/input.csv",
    timeout=600,
    dag=dag
)

Important Hook and Sensor Methods

  • hook.get_conn(): Returns a connection to an external service.
  • hook.run(sql): Executes a SQL query (for database hooks).
  • sensor.poke(context): Checks if the condition is met.
  • sensor.execute(context): Runs the sensor until the condition is met.

Installing and Running Airflow

To install Airflow:

pip install apache-airflow

To start the Airflow web server and scheduler:

airflow standalone

The UI will be available at http://localhost:8080.

Task Dependencies and Execution Order

Tasks can be arranged using >> (sequential execution) or << (parallel execution):

task1 >> task2  # task2 runs after task1
task3.set_upstream(task1)  # Equivalent to task1 >> task3
task4.set_downstream(task5)  # Equivalent to task4 >> task5

Monitoring and Debugging Workflows

  • Web UI: Provides a visual representation of DAGs and task execution.
  • Logs: Available for each task in the UI.
  • Retries: Automatically re-run failed tasks with exponential backoff.

Scaling Airflow

For larger workflows, use: - CeleryExecutor: Distributed execution using Celery workers. - KubernetesExecutor: Dynamically schedules tasks in Kubernetes. - Database-backed queues: Store execution state in PostgreSQL/MySQL instead of SQLite.

Conclusion

Apache Airflow is a powerful workflow orchestration tool suitable for data engineering, ETL, and machine learning pipelines. By using DAGs, operators, hooks, and executors, you can automate complex workflows efficiently.

Would you like an advanced example using cloud services like AWS or GCP? Let me know!