Updated June 2026

Distributed Task Automation with Python Celery and RabbitMQ

Class Duration

16 hours of live training delivered over 2-4 days.

Student Prerequisites

All students should have taken the Python Task Automation course or have significant experience with the topics covered in the Python Task Automation course.
Experience with RabbitMQ and PostgreSQL is helpful but not required.

Target Audience

Python developers and programming professionals who want to automate complex, distributed workflows and boost backend efficiency by building, monitoring, and deploying scalable task queues with Celery and RabbitMQ. This course builds on the fundamentals from Task Automation with Python.

Description

This course gives Python developers hands-on mastery in distributed task automation using Celery and RabbitMQ. Beginning with environment setup in Visual Studio Code, leveraging Python and Docker, it covers containerization principles and remote data storage, guiding you through running PostgreSQL and RabbitMQ in containers. You'll gain hands-on experience with Celery from foundational concepts to advanced techniques like routing and scheduling, learning to build, monitor, and deploy robust task queues. The course concludes with practical strategies for scaling and deploying Celery in production. Whether you're a seasoned developer or new to programming, this customizable course equips you with the skills to automate complex, distributed workflows and boost backend efficiency, with real-world examples relevant to your domain.

Learning Outcomes

Understand the concept of Distributed Task Automation and learn about tools like Celery and RabbitMQ.
Set up a development environment with Visual Studio Code, Python, and Docker.
Learn about containerization using Docker and Docker Compose.
Understand remote data storage and how to run PostgreSQL in a container.
Learn about Message Brokers and how to run RabbitMQ in a container.
Gain a solid understanding of Celery basics, including task definition, running tasks, and worker management.
Delve into advanced Celery concepts like task serialization, routing, and scheduling periodic tasks.
Learn how to scale and deploy Celery in a production environment, including load balancing, high availability, and performance tuning.

Training Materials

Comprehensive courseware is distributed online at the start of class. All students receive a downloadable MP4 recording of the training.

Software Requirements

Students will need a free, personal GitHub account to access the courseware. Student machines will need a text editor like Visual Studio Code, the latest Python version, Docker Desktop, PanDoc, and OpenOffice. Students will need permission to install NPM and PyPi packages as well as the ability to download Docker images. Preconfigured student virtual machines can be provided upon request.

Training Topics

Overview of Distributed Task Automation

What is Distributed Task Automation?
Overview of Celery and its Features
Celery compared to Faust
What is Message Broker?
What is RabbitMQ?
RabbitMQ + PostgreSQL compared to Kafka + Zookeeper

Development Environment

Configure Visual Studio Code for Python Script Programming
Python Code Linting & Reformatting with Ruff & MyPy
Debugging Python Scripts with Visual Studio Code
Docker Desktop

Containerization

What is a Container?
What is Docker?
What is Docker Hub?
Images and Containers
Create an Image with Dockerfile
Run Containers
Configure Containers with Environment Variables
Docker Compose
Docker Compose Networking
Docker Compose Volume

Remote Data Storage

What is Remote Data Storage?
Running a Data Storage in a Container
Running PostgreSQL in a Container
Running PostgreSQL Client Tool in a Container
Configure with Docker Compose
Read/Write Data to PostgreSQL with Python SQLAlchemy

Remote Message Broker

What is a Message Broker?
Running a Message Broker in a Container
Running RabbitMQ in a Container
Running RabbitMQ Client Tool in a Container
Configure with Docker Compose
Read/Write Data to RabbitMQ with Python and Pika

Celery Fundamentals

The Celery Application: Configuration and Setup
Defining Tasks with Decorators
Calling Tasks: delay, apply_async, and Signatures
Brokers and Result Backends: RabbitMQ + PostgreSQL
Running and Managing Workers
Typed Task Payloads with Pydantic Models (Celery 5.5+)

Task Design and Reliability

Idempotent Task Design
Retries, Retry Backoff, and Time Limits
Task Serialization Options
Late Acknowledgement (acks_late) and Redelivery
Error Handling and Dead-Letter Patterns

Routing and Workflows

Queues, Exchanges, and Routing Keys
Routing Tasks to Dedicated Queues and Workers
Canvas: Chains, Groups, and Chords
Scheduling Periodic Tasks with Celery Beat

Monitoring and Operations

Monitoring Workers and Tasks with Flower
Inspecting Queues with the RabbitMQ Management UI
Worker Concurrency, Prefetch, and Autoscaling
Soft Shutdown and Graceful Worker Lifecycle (Celery 5.5+)

Scaling and Production Deployment

Containerizing Celery Workers with Docker Compose
Scaling Workers Horizontally
High Availability for RabbitMQ
Performance Tuning and Capacity Planning