Docker容器的kafka在VM虚拟机挂起重新运行之后连接异常解决

Docker容器的kafka在VM虚拟机挂起重新运行之后连接异常解决

在虚拟机安装Docker形式的kafka可参考：
https://blog.csdn.net/weixin_42949219/article/details/146045338?spm=1011.2415.3001.5331
当时可以连接，但是在虚拟机挂起重新运行之后连接异常：我估计应该是IP动态变化导致的，所以需要修改对应配置。

以下是我的解决方案：

docker-compose.yml文件

version: '3.8'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - kafka-net
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_LISTEN_ADDRESS: 0.0.0.0  # 关键：强制监听所有接口
    volumes:
      - ./zookeeper/data:/data           # 数据持久化
    healthcheck:
      test: ["CMD-SHELL", "echo stat | nc 127.0.0.1 2181 | grep Mode"]
      interval: 10s
      timeout: 5s
      retries: 3

  kafka1:
    image: wurstmeister/kafka
    container_name: kafka1
    ports:
      - "19092:9092"
    networks:
      - kafka-net
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181  # 使用服务名称访问
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://${HOST_IP}:19092  # 动态IP
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    volumes:
      - ./kafka1/data:/opt/kafka/data  # 数据持久化
    depends_on:
      zookeeper:
        condition: service_healthy  # 等待ZooKeeper就绪

  kafka2:
    image: wurstmeister/kafka
    container_name: kafka2
    ports:
      - "19093:9092"
    networks:
      - kafka-net
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9093,OUTSIDE://${HOST_IP}:19093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    volumes:
      - ./kafka2/data:/opt/kafka/data
    depends_on:
      zookeeper:
        condition: service_healthy

  kafka3:
    image: wurstmeister/kafka
    container_name: kafka3
    ports:
      - "19094:9092"
    networks:
      - kafka-net
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9093,OUTSIDE://${HOST_IP}:19094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    volumes:
      - ./kafka3/data:/opt/kafka/data
    depends_on:
      zookeeper:
        condition: service_healthy

networks:
  kafka-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/16  # 固定子网防止IP变化

操作步骤

清理旧环境

docker-compose down -v  # 清除旧数据和网络
rm -rf zookeeper/data kafka*/data  # 删除残留数据（首次可跳过）

设置宿主机IP变量

export HOST_IP=$(hostname -I | awk '{print $1}')  # 自动获取本机IP

启动服务

docker-compose up -d

验证服务状态

检查ZooKeeper日志

docker logs zookeeper | grep "binding to port"

检查Kafka节点日志

docker logs kafka1 | grep "Registered broker 1"

测试跨容器通信

# 安装网络工具
apt update && apt install -y telnet

docker exec -it kafka1 telnet zookeeper 2181

创建测试Topic

docker exec kafka1 kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic test-topic \
  --partitions 3 \
  --replication-factor 2

关键配置说明

网络稳定性优化
显式指定子网 (172.20.0.0/16) 防止虚拟机恢复时IP变化
使用 condition: service_healthy 确保依赖启动顺序

ZooKeeper 可靠性
ZOOKEEPER_LISTEN_ADDRESS: 0.0.0.0 强制监听所有接口
健康检查通过 nc 命令验证服务状态

Kafka 动态IP配置
${HOST_IP} 环境变量动态注入宿主机IP
每个Broker的 OUTSIDE 端口递增映射 (19092-19094)

数据持久化
ZooKeeper数据目录: ./zookeeper/data
Kafka数据目录: ./kafka{1-3}/data

常见问题处理

持续出现 ZooKeeper 连接超时

检查容器IP是否在同一个子网
docker network inspect kafka-net | grep IPv4Address

临时关闭SELinux/AppArmor
sudo setenforce 0  # CentOS/RHEL
sudo systemctl stop apparmor  # Ubuntu

Topic创建失败

检查副本数是否小于等于Broker数量
docker exec kafka1 kafka-topics.sh --describe --topic test-topic --bootstrap-server localhost:9092

外部客户端无法连接

 验证宿主机防火墙规则
sudo iptables -L -n | grep 1909
telnet ${HOST_IP} 19092

该配置通过动态IP绑定、网络子网固定和数据持久化，可有效应对虚拟机挂起恢复后的网络变化问题。