Concourse to RabbitMQ Pipeline Migration

The shift

We’re moving a Go-based pipeline off Concourse and onto RabbitMQ — from CI-orchestrated jobs toward a message-driven model where work is queued, consumed, and retried with explicit failure semantics.

Why RabbitMQ

Concourse served us well for orchestration, but this workload benefits from durable queues, consumer scaling, and first-class messaging patterns. RabbitMQ gives us a clearer separation between “enqueue work” and “process work,” which fits how the Go services already think about the problem domain.

Dead-letter exchanges

A core part of the design is dead-letter exchange (DLX) routing — messages that fail processing or exceed retry policy land somewhere inspectable instead of disappearing. That supports debugging, reprocessing, and alerting without blocking the happy path.

Poison messages get quarantined. Transient failures get retried with bounds. Operators can see what’s stuck and why.

ADR approved

The architecture went through formal Architecture Decision Record (ADR) review — and it’s approved. Having that documentation locked in means implementation can proceed with shared understanding of queue topology, failure handling, compatibility constraints, and rollback thinking.

What I’m focused on now

Translating the ADR into incremental rollout: producers publishing safely, consumers idempotent, observability on queue depth and DLQ volume, and a migration path that doesn’t require a single maintenance window for everything at once.

Tie-in to earlier work

This connects directly to prior RabbitMQ video processing experience — validation, serialization, quorum queues, and cross-team coordination — applied now at the platform pipeline layer in Go.