Learning Kafka

If RabbitMQ is a kitchen ticket rail — take an order, finish it, move on — Kafka is more like a security camera recording that never stops.

Events get appended to a log. Anyone can come back later and watch from wherever they left off. Nothing disappears just because someone already read it.

I am learning Kafka separately. When a project actually needs that “recording” model, I want to recognize it early — not after the team has already built the wrong thing.

The simple idea

Queue mindset: deliver a task, mark it done, gone.

Log mindset: record what happened, keep the tape, let many readers catch up at their own speed.

flowchart TB
  subgraph queue["Queue style — task is done and gone"]
    Q1[Job] --> W[Worker finishes]
    W --> X[Job removed]
  end

  subgraph log["Log style — history stays"]
    E1[Event] --> L[Append to log]
    L --> R1[Reader A at their pace]
    L --> R2[Reader B at their pace]
    L --> R3[Reader C rewinds and replays]
  end

That replay part is the superpower. Need to rebuild a dashboard from last Tuesday? Re-read the log. New service wants the same stream of facts? Attach another reader — you do not re-send everything by hand.

When Kafka fits

Good fits:

• Audit trails — “show me everything that happened to this account”
• Analytics pipelines — many teams reading the same firehose of events
• Systems that need rewind — new app version replays history to warm up

Less ideal when you just want one worker to pick up the next job and ack it. That is queue territory — RabbitMQ, SQS, etc.

Neither is universally better. They answer different questions.

flowchart LR
  P[Producer writes events] --> K[Kafka log]
  K --> C1[Consumer group A]
  K --> C2[Consumer group B]
  K --> C3[New service replays from start]

Consumer group — fancy term for “a team of readers sharing the load.” Each group tracks its own progress through the tape.

How I am learning

This one is background depth for now — reading, local experiments, comparing feel to queue-based patterns I am already learning.

When a problem needs durable history or multiple independent readers, that is my cue to go deeper.

Hands-on lab: Event Patterns — RabbitMQ and Kafka running side by side so I can see the difference instead of reading about it.

Where this goes

I want to walk into design chats and know which shape fits: hand off a task vs. preserve a story. And I want to understand terms like partitions and offsets well enough to ask good questions — not nod along.

That is it. No rush. This track turns on when the problem needs a recording, not a ticket.