Question

I am trying to run a single Kafka instance in docker. I can create topics but cannot consume from or produce to them. When I try my Kafka broker logs an error message.

kafka_1 | [2019-03-24 16:45:05,263] ERROR [KafkaApi-5] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

I don't understand the Number of alive brokers '0' bit as it is a Kafka broker that is logging this so surely there must be at least one broker (itself)?

I am using the default server.properties file with the following changes:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.0.109:9092
zookeeper.connect=zookeeper:2181

where 192.168.0.109 is my host machine IP.

Files

docker-compose

version: '3.4'
services:
 zookeeper:
 build:
 context: ./
 dockerfile: zookeeper.dockerfile
 network: host
 image: zookeeper
 ports:
 - "2181:2181"
 kafka:
 build:
 context: ./
 dockerfile: kafka.dockerfile
 network: host
 image: kafka
 ports:
 - "9092:9092"

kafka.dockerfile

FROM alpine:3.7

# Set the name for a non-priliaged user
ENV APPUSER=appuser

# Install dependnacies
RUN apk add bash
RUN apk add openjdk8-jre

# Add a non-privilaged user and change into it's home directory
RUN adduser -D $APPUSER
WORKDIR /home/$APPUSER

# Download kafka and set its owner to $APPUSER
RUN wget https://www-eu.apache.org/dist/kafka/2.1.0/kafka_2.11-2.1.0.tgz
RUN chown $APPUSER kafka_2.11-2.1.0.tgz

# Change to the non-privilaged user
USER $APPUSER

# Extract kafka
RUN tar -xzf kafka_2.11-2.1.0.tgz

# copy custom server.properties into image 
COPY ./server.properties ./

# Set working directory to kafka dir
WORKDIR /home/$APPUSER/kafka_2.11-2.1.0

# start kafka with customer server.properties file
CMD ["bin/kafka-server-start.sh", "../server.properties"]

zookeeper.dockerfile

FROM alpine:3.7

# Set the name for a non-priliaged user
ENV APPUSER=appuser

# Install dependnacies
RUN apk add bash
RUN apk add openjdk8-jre

# Add a non-privilaged user and change into it's home directory
RUN adduser -D $APPUSER
WORKDIR /home/$APPUSER

# Download kafka and set its owner to $APPUSER
RUN wget https://www-eu.apache.org/dist/kafka/2.1.0/kafka_2.11-2.1.0.tgz
RUN chown $APPUSER kafka_2.11-2.1.0.tgz

# Change to the non-privilaged user
USER $APPUSER

# Extract kafka
RUN tar -xzf kafka_2.11-2.1.0.tgz

# Set working directory to kafka dir
WORKDIR /home/$APPUSER/kafka_2.11-2.1.0

# Run zookeeper
CMD ["bin/zookeeper-server-start.sh", "config/zookeeper.properties"]

This is down to the listeners configuration.

When you specify advertised.listeners=PLAINTEXT://192.168.0.109:9092 then Kafka will try to connect to 192.168.0.109 from within the Docker container. But if this is a host address, and if you've not set up the appropriate Docker networking wizardry, it will fail.

When you specify advertised.listeners=PLAINTEXT://localhost:9092 then Kafka will connect to localhost within the Docker container, which of course resolves to itself. The catch here is that you would be unable to use a Kafka client outside of the Docker container (either on your host machine, or another Docker container) because they would try to connect to localhost—locally to themselves on which there would not be a Kafka broker.

This article explains in more detail the issue, and appropriate configuration to use. If you want a pre-built example of Kafka in Docker then this Docker Compose shows it.