大数据环境（单机版） Flume传输数据到Kafka

前言

flume监控指定目录，传输数据到kafka

一、准备

flume-1.10.1
kafka_2.11-2.4.1
zookeeper-3.4.13

二、安装

使用命令 tar -zxf 分别解压到指定目录

三、配置环境变量

vi ~/.bash_profile

export FLUME_HOME=/home/hadoop/install/flume-1.10.1
export PATH=$FLUME_HOME/bin:$PATH

export ZOOKEEPER_HOME=/home/hadoop/install/zookeeper-3.4.13
export PATH=$ZOOKEEPER_HOME/bin:$PATH

export KAFKA_HOME=/home/hadoop/install/kafka_2.11-2.4.1
export PATH=$PATH:$KAFKA_HOME/bin

# 使配置生效
source ~/.bash_profile

四、修改配置

4.1、kafka配置

修改zk地址

vi $KAFKA_HOME/config/server.properties

修改zk的连接地址
zookeeper.connect=localhost:2181

4.2、Flume配置

拷贝配置

cp $FLUME_HOME/config/flume-conf.properties.template $FLUME_HOME/config/flume-conf.properties

编辑配置

vi $FLUME_HOME/config/flume-conf.properties

增加如下配置

# 增加如下配置
a1.sources = r1
a1.channels = c1

a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
# 这个目录是 flume监控的目录
a1.sources.r1.filegroups.f1 = /home/hadoop/install/temp/flume/applog/log/app.*
a1.sources.r1.positionFile = /home/hadoop/install/temp/flume/json/taildir_position.json

a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = localhost:9092
# kafka的主题
a1.channels.c1.kafka.topic = topic_log
# 不把数据解析为 Flume的Event，kafka原样接收
a1.channels.c1.parseAsFlumeEvent = false

a1.sources.r1.channels = c1

五、启动程序

5.1、启动zk

zkServer.sh start

5.2、启动kafka

sh kafka-server-start.sh /home/hadoop/install/kafka_2.11-2.4.1/config/server.properties &

5.3、启动flume

flume-ng agent -c conf -f conf/flume-netcat-conf.properties -n a1 -Dflume.root.logger=INFO,console &

六、测试

6.1、启动一个kafka终端，用来消费消息

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_log

6.2、写入日志

echo {"a":"b1"} >> /home/hadoop/install/temp/flume/applog/log/app.log
echo {"a":"b2"} >> /home/hadoop/install/temp/flume/applog/log/app.log

其他

软件下载