kafka 3.5.0 raft协议安装
前言
最近做项目，需要使用kafka进行通信，且只能使用kafka，笔者没有测试集群，就自己搭建了kafka集群，实际上笔者在很早之前就搭建了，因为当时还是zookeeper（简称ZK）注册元数据，现在新版kafka（3.0.0开始）已经自带了元数据能力（使用raft协议）减少了kafka对zk的依赖性。笔者在查询资料发现，说jdk至少jdk11，实测jdk8也能运行，且并不需要网上说的3+4节点，3+3即可，当然理论上broker节点越多越好，但是元数据节点建议3、5个最合适，raft的过半一致性和容错性的综合取舍。
准备
准备kafka安装包：Apache Kafka
笔者使用的kafka 3.5.0和scala 2.13，采用3台虚拟机，当然容器也不是不行，注意持久化pv pvc和配置的管理（ip换成域名，dns的切换支持），中间件建议使用虚拟机，可以降低很多容错性。
jdk使用open jdk，配置java_home和path，以Ubuntu为例
sudo apt install openjdk-8-jdk-headless
以macOS为例，创建一个ubuntu-server 最小安装的虚拟机（vmware，毕竟个人使用不要钱），然后安装openssh 和 openjdk，然后shutdown now
网络选择桥接，相当于一台“真实在”网络上的一台物理机
这样就得到了
192.168.0.108
192.168.0.107
192.168.0.106
3台虚拟机
步骤
先看kafka集群的架构图，实际上安装的过程就是架构图的执行过程
从图中可以看出已经没有zk的存在了，从kafka节点自己管理元数据，通过raft协议选主的方式。
1. kafka的准备
上传kafka安装包，必须是二进制安装包，不要源码包，编译比较麻烦，然后解压
tar -zxvf kafka_2.13-3.5.0.gz
查看配置目录会发现
明显多了kraft的配置目录，那么如果使用kafka raft元数据中心，则需要修改kraft目录，启动时指定kraft目录的配置
2. 配置修改
raft协议实际上跟zk差不多，使用raft协议的中间件就太多了，但是本质上每个节点都需要一个唯一id，zk也是如此，所以kafka kraft就相当于集成的zk。
在kraft下的有3个文件文件，其中启动相关的是server.properties中
执行配置修改
# The role of this server. Setting this puts us in KRaft mode
process.roles=broker,controller
# The node id associated with this instance's roles
node.id=1
# The connect string for the controller quorum
controller.quorum.voters=1@localhost:9093
############################# Socket Server Settings #############################
# The address the socket server listens on.
# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
# Name of listener used for communication between brokers.
inter.broker.listener.name=PLAINTEXT
# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://localhost:9092
# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.names=CONTROLLER
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/tmp/kraft-combined-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
每一行都有注释，重点关注
笔者设定
192.168.0.106 nodeid 1
192.168.0.107 nodeid 2
192.168.0.108 nodeid 3
至此配置基本上完成，同理一个节点可以同时是controller和broker，也可以仅仅是controller或者broker，因为controller的负载比较轻，所以一般是和broker一起。其中有个log.dir这个的路径是下面元数据生成的路径（选主）和数据事务日志，索引日志的存储目录
3. 启动
1. 生成uuid
任意找一个节点执行：
./kafka-storage.sh random-uuid
每次执行uuid会不一样，这个uuid标识是一个集群，所以所有节点公用一个uuid，不要每个节点重新生成，会识别不了
然后执行format，如下标红是我生成的，这个每次不是固定的
./kafka-storage.sh format -t gZzkfRm4T1y8wSAY-ZNG5Q -c ../config/kraft/server.properties
格式化配置文件，同步其他节点
配置文件有什么变化？在日志配置的目录下出现
关键还是meta的文件，有集群id和节点id，版本号，这个对启动至关重要。
即在上面的log.dir的目录生成，所以尽量不能使用临时目录
2. 启动
启动就很简单了，使用刚刚配置的server.properties执行启动即可
./kafka-server-start.sh -daemon ../config/kraft/server.properties
不过为了方便查看启动日志，建议执行日志的console文件输出
先看事务日志和索引
验证
验证很简单，查看bin同级目录下的日志即可
日志带有[2025-02-08 08:34:12,286] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)
如果生成用途可以安装kafka的控制台，kafka-ui，不过我这里就不安装了，因为docker安装比较容易。
总结
kafka从3.0.0开始推出了raft模式的元数据中心，实际上类似zk，kafka自己命名kraft。使用这种方式搭建kafka集群将不再需要zk，同理，kafka的集群的每个节点可以同时是broker和controller（以前zk充当），也可以是单独的broker，controller（负载不重，不建议单独controller，跟zk没区别），官方说明需要jdk11及以上，实测jdk8可以运行，但是生成建议严格按照官方标定的jdk执行，jdk是向下兼容的，但是不确定是否会涉及新api或新特性的使用。
另外实际使用中，可能会涉及使用iptables做nat限制kafka的连接方，比如在kafka节点通过iptables限制发送者或者消费端的ip
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 9093 -j DNAT --to-destination kafkaxxx:9093
kafkaxxx --- 指定的是 Kafka 服务所在的机器地址
如果kafka是对接方提供，则在nat打通时，需要客户端连接的服务器也执行iptables，否则可能出现连接kafka正常，但是不能消费。
iptables -t nat -A POSTROUTING -p tcp -m tcp --dport 9093 -j SNAT --to-source natxxx
natxxx --- 指定的是配置 iptables 的本机的地址