Updated June 2026

Apache Airflow Administration: Scalable Workflow Automation and Orchestration

Class Duration

14 hours of live training delivered over 2-4 days.

Student Prerequisites

Practical experience writing Python programs
Basic Linux command line skills
Familiarity with containerization and Kubernetes concepts is helpful

Target Audience

Platform engineers, DevOps professionals, system administrators, and data engineers responsible for deploying, configuring, securing, and scaling Apache Airflow 3 in production environments, including Kubernetes-based deployments.

Description

This intensive course delivers a deep dive into Apache Airflow 3's architecture and core services—the api-server, scheduler, dag processor, triggerer, and workers—while contrasting it with Cron Jobs and Celery. Through hands-on labs in installation, Python/PostgreSQL and Kubernetes (EKS/Helm) deployment, custom container image building, upgrading from Airflow 2 to Airflow 3, and monitoring with logs, OpenTelemetry metrics, and Grafana, participants will master the skills to configure, secure, scale and optimize production-grade workflow automation solutions.

This course provides a deep dive into Apache Airflow 3, a powerful workflow automation platform for managing complex data pipelines. Participants will explore the architecture of Airflow 3, including its core services—the api-server, scheduler, dag processor, triggerer, and workers—along with Directed Acyclic Graphs (DAGs), operators, and executors. The course covers installation, configuration, auth managers, upgrading from Airflow 2 to Airflow 3, and integration with Kubernetes, AWS EKS, and Helm. Attendees will gain hands-on experience deploying Airflow, optimizing workflows, customizing container images, and monitoring performance using logging and OpenTelemetry metrics. Designed for professionals, this course ensures participants can build scalable, reliable, secure, and efficient workflow automation solutions.

Learning Outcomes

Understand Apache Airflow 3's architecture—api-server, scheduler, dag processor, triggerer, and workers—and how it compares to Cron Jobs and Celery.
Learn the fundamentals of DAGs, operators, tasks, variables, assets, and schedulers.
Install and configure Apache Airflow using Python environments, PostgreSQL, and Kubernetes, and plan upgrades from Airflow 2 to Airflow 3.
Gain hands-on experience deploying Airflow on Kubernetes, including EKS and Helm.
Configure Airflow's executors—Local, Celery, Kubernetes, and Edge—plus auth managers, logs, and advanced settings for scalability and security.
Build and use custom Airflow container images with additional dependencies.
Implement monitoring solutions using logs, OpenTelemetry metrics, Grafana, and external storage.
Apply best practices for workflow reliability, scaling, security, and automation in production environments.

Training Materials

Comprehensive courseware is distributed online at the start of class. All students receive a downloadable MP4 recording of the training.

Software Requirements

Students will need a free, personal GitHub account to access the courseware. Students will need permission to install Python and Visual Studio Code on their computers. Also, students will need permission to install Python Packages and Visual Studio Code extensions. If students are unable to configure a local environment, a cloud-based environment can be provided.

Training Topics

What is Apache Airflow?

Distributed Task Automation
Compared to Cron Jobs
Compared to Celery
Scalability and Reliability
Directed Acyclic Graphs (DAGs)
Workflows as Code
What's New in Airflow 3
Airflow 3 Services: API Server, Scheduler, DAG Processor, Triggerer, and Workers

Workflows as Code (no programming)

Anatomy of a DAG
Directed Acyclic Graphs
Operators
Tasks
Variables
XComs
Providers
Connections
Assets (formerly Datasets)
DAG Versioning
Explore how DAG parts connect to the new UI
DAG Serialization
Schedulers
Pools

Installation and Configuration

Python Virtual Environment
Install Airflow
Airflow Constraints File
Standalone Mode
Run the API Server, Scheduler, DAG Processor, and Triggerer Independently
SQLite vs PostgreSQL
Configure with PostgreSQL
Airflow and Kubernetes (with Minikube)
Airflow and AWS Elastic Kubernetes Service (EKS)
Airflow Helm Chart

Hands-On Kubernetes (K8s)

Containerization and Orchestration
Kubectl
Helm
Nodes
Namespaces
Pods, Containers, and Services
Connect to the Internet (EKS)
Keda Autoscaler
Pod Logs
SSH into Pods/Containers
Live Upgrading Airflow
Upgrading from Airflow 2 to Airflow 3

Airflow Configuration

Airflow Configuration File Location
Airflow Executor Configuration
Airflow Log Levels
Helm Chart Configuration
Learn How to Configure Airflow and K8s Pods
Local Executor
Celery Executor
Kubernetes Executor
Edge Executor
Auth Managers (SimpleAuthManager and FAB)

Airflow Custom Image

Airflow Container Image
Why Create a Custom Image?
Create a Custom Image
Install Software with Apt
Install Software with PyPi
Install Providers and Custom Software
Use the Custom Container Image

Monitoring

Logging
Log File Structure
Log Levels
Review Task Logs in the UI
External Log Storage
OpenTelemetry Metrics and Traces
Monitor with Grafana
Notifications and Deadline Alerts