apache druid整合hadoop3.3

安装流程

修改配置文件

1. 解压文件

   tar -xvf apache-druid-31.0.0-bin.tar.gz
   mv apache-druid-31.0.0 druid

2. 拷贝hadoop配置文件到druid

   ln -s /app/hadoop-3.3.6/etc/hadoop/core-site.xml /app/druid/conf/druid/cluster/_common/core-site.xml
   ln -s /app/hadoop-3.3.6/etc/hadoop/hdfs-site.xml /app/druid/conf/druid/cluster/_common/hdfs-site.xml
   ln -s /app/hadoop-3.3.6/etc/hadoop/yarn-site.xml /app/druid/conf/druid/cluster/_common/yarn-site.xml
   ln -s /app/hadoop-3.3.6/etc/hadoop/mapred-site.xml /app/druid/conf/druid/cluster/_common/mapred-site.xml

3. 拷贝连接MYSQL的jar包

   将mysql的jdbc驱动拷贝到/app/druid/extensions/mysql-metadata-storage

4. 添加druid的mysql用户

   create database druid;
   create user ‘druid’@‘%’ identified by ‘druid’;
   ALTER USER ‘druid’@‘%’ IDENTIFIED BY ‘druid’;
   grant all privileges on druid.* to ‘druid’@‘%’ with grant option;
   grant all privileges on mysql.* to ‘druid’@‘%’ with grant option;

5. 编辑druid的配置文件

   vim /app/druid/conf/druid/cluster/_common/common.runtime.properties

   druid.extensions.loadList=[“mysql-metadata-storage”,“druid-hdfs-storage”, “druid-kafka-indexing-service”, “druid-datasketches”, “druid-multi-stage-query”, “druid-parquet-extensions”]

   druid.host=hadoop-drill-nn-1

   druid.zk.service.host=10.10.9.22:2181
   druid.zk.paths.base=/druid

   druid.metadata.storage.type=mysql
   druid.metadata.storage.connector.connectURI=jdbc:mysql://10.10.100.95:3306/druid
   druid.metadata.storage.connector.user=druid
   druid.metadata.storage.connector.password=druid

   druid.storage.type=hdfs
   druid.storage.storageDirectory=/druid/segments
   druid.indexer.logs.type=hdfs
   druid.indexer.logs.directory=/druid/indexing-logs

6. 修改coordinator-overlord配置（主节点）

   vim /app/druid/conf/druid/cluster/master/coordinator-overlord/jvm.config

   -server
   -Xms2g
   -Xmx2g
   -XX:+ExitOnOutOfMemoryError
   -XX:+UseG1GC
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=var/tmp
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   -Dderby.stream.error.file=var/druid/derby.log

   vim /app/druid/conf/druid/cluster/master/coordinator-overlord/runtime.properties

   druid.coordinator.http.port=18082

7. 修改historical配置（数据节点）

   vim /app/druid/conf/druid/cluster/data/historical/jvm.config

   -server
   -Xms2g
   -Xmx2g
   -XX:MaxDirectMemorySize=13g
   -XX:+ExitOnOutOfMemoryError
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=var/tmp
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

   vim /app/druid/conf/druid/cluster/data/historical/runtime.properties

   #内存不够减小此值
   druid.processing.buffer.sizeBytes=50MiB
   druid.historical.http.port=18083

8. 修改middleManager（数据节点）

   vim /app/druid/conf/druid/cluster/data/middleManager/jvm.config

   -server
   -Xms128m
   -Xmx128m
   -XX:+ExitOnOutOfMemoryError
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=var/tmp
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

   vim /app/druid/conf/druid/cluster/data/middleManager/runtime.properties

   druid.indexer.runner.http.port=18091

9. 修改broker（查询节点）

   vim /app/druid/conf/druid/cluster/query/broker/jvm.config

   -server
   -Xms1g
   -Xmx1g
   -XX:MaxDirectMemorySize=6g
   -XX:+ExitOnOutOfMemoryError
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=var/tmp
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

   vim /app/druid/conf/druid/cluster/query/broker/runtime.properties

   druid.processing.buffer.sizeBytes=50MiB
   druid.broker.http.port=18082

10. 修改router（查询节点）

    vim /app/druid/conf/druid/cluster/query/router/jvm.config

    vim /app/druid/conf/druid/cluster/query/router/runtime.properties
    druid.router.http.port=18888

启动相关进程

1. 启动查询节点

   nohup start-cluster-query-server &

2. 启动数据节点

   nohup start-cluster-data-server &

3. 启动主节点

   nohup start-cluster-master-no-zk-server &

停止服务

/app/druid/bin/service --down

查看运行情况

http://10.10.100.95:8888