Wiring RedPanda for Local Kafka Dev

Because taming Kafka shouldn’t feel like working at the Kafka Zoo.

You know the drill. You’re testing a Kafka producer. Or maybe you’re validating a stream processor that does clever things with Protobuf, Avro, or your very own half-baked JSON schema. You fire up your Docker Compose stack, wait for Kafka and Zookeeper to hold hands… and then everything breaks because your client can’t find the advertised broker address. Again.

Enter Redpanda: a Kafka API-compatible, Zookeeper-less, single-binary, developer-friendly delight. It’s like switching from a clingy elephant to a caffeinated panda that actually reads the dev docs.

Here’s why Redpanda deserves a spot in your local dev toolbox—and the one tiny configuration hiccup that’ll haunt you until you fix it.

TL;DR – Just Give Me the Compose File

services:

  kafka:
    container_name: kafka
    image: redpandadata/redpanda:latest
    command:
      - redpanda start
      - --smp 1
      - --mode dev-container
      - --kafka-addr PLAINTEXT://0.0.0.0:9092,OUTSIDE://0.0.0.0:9093
      - --advertise-kafka-addr PLAINTEXT://kafka:9092,OUTSIDE://localhost:9092
    ports:
      - "9092:9093"
      - "9644:9644"
    volumes:
      - kafka:/var/lib/redpanda/data

  kafka-ui:
    container_name: kafka-ui
    image: redpandadata/console:latest
    environment:
      KAFKA_BROKERS: "kafka:9092"
    ports:
      - "8080:8080"
    depends_on:
      - kafka

volumes:
  kafka:

This Compose file sets up:

A Redpanda node (kafka) that:
- accepts connections on localhost:9092 from your host machine
- accepts connections on kafka:9092 from inside the Docker network
A Kafka UI that connects happily to the internal hostname kafka:9092

But look closer - like "Why isn't my client connecting?" closer.

The Gotcha: `--kafka-addr` vs `--advertise-kafka-addr`

Redpanda supports multiple listener addresses: one for accepting connections and another for advertising to clients. This is both genius and maddening.

Here’s what these flags are doing:

--kafka-addr PLAINTEXT://0.0.0.0:9092,OUTSIDE://0.0.0.0:9093

Binds two listeners:
- PLAINTEXT on 0.0.0.0:9092 (used inside the Docker network)
- OUTSIDE on 0.0.0.0:9093 (used by your host machine)

--advertise-kafka-addr PLAINTEXT://kafka:9092,OUTSIDE://localhost:9092

Tells Redpanda what hostnames to advertise to clients:
- PLAINTEXT resolves to kafka:9092 from within Docker
- OUTSIDE resolves to localhost:9092 from your machine

Notice the mapping trick: we expose internal port 9092 outside via host port 9092 by binding container port 9093:

ports:
  - "9092:9093"

This is counterintuitive. Yes, it's 9092 on your host, but 9093 inside the container. Think of it as a reverse mullet: business on the outside, party on the inside.

Sanity Check: Does It Work?

Try it with the kafkacat CLI (now kcat, but I'm sentimental):

docker run --rm --net=host edenhill/kafkacat -P -b localhost:9092 -t hello-world

Or from within Docker:

docker exec -it kafka-ui curl kafka:9092

Your browser at http://localhost:8080 should give you a snappy Redpanda UI. You'll feel like an observant zookeeper.

Why Bother?

Redpanda is perfect for:

Writing client libraries – quick restart, no Zoo.
Trying out Kafka clients in different languages – Go, Rust, Python, even that one guy using OCaml.
Teaching, testing, or torturing Kafka without overhead.

It's small. It's fast. It doesn't call its mom every time it boots up.

Final Thoughts

This setup is my go-to for developing Kafka-ish things in peace. Redpanda lets me iterate quickly without worrying about sidecars, zookeepers, or DNS gymnastics.