当前位置：首页 > article >正文

搭建cdh集群及问题处理

article 2024/11/17 4:50:57

CM安装部署
1、MySQL中建库-创建各组件需要的数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
show databases;
±-------------------+
| Database |
±-------------------+
| hive |
| hue |
| information_schema |
| mysql |
| oozie |
| performance_schema |
| scm |
| sys |
±-------------------+

2、Cloudera Manager安装
下载mysql-connector-java.jar https://downloads.mysql.com/archives/c-j/
拷贝到/usr/share/java\路径下，并重命名
tar -zxvf mysql-connector-java-8.0.20.tar.gz
cd mysql-connector-java-8.0.20/
mkdir /usr/share/java
cp mysql-connector-java-8.0.20.jar /usr/share/java/mysql-connector-java.jar
分发 ./dep.sh /usr/share/java/mysql-connector-java.jar /usr/share/java/ all

集群规划
node1 cloudera-scm-server/cloudera-scm-agent
node2 cloudera-scm-agent
node3 cloudera-scm-agent

卸载残留
rpm -qa | grep cloudera
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64
cloudera-manager-agent-6.3.1-1466458.el7.x86_64

rpm -e cloudera-manager-daemons-6.3.1-1466458.el7.x86_64 --nodeps
rpm -e cloudera-manager-agent-6.3.1-1466458.el7.x86_64 --nodeps

安装CM
yum list | grep cloudera
cloudera-manager-agent.x86_64 6.3.1-1466458.el7 cloudera-manager
cloudera-manager-daemons.x86_64 6.3.1-1466458.el7 cloudera-manager
cloudera-manager-server.x86_64 6.3.1-1466458.el7 cloudera-manager
cloudera-manager-server-db-2.x86_64 6.3.1-1466458.el7 cloudera-manager

安装cloudera-manager-agent
./cmd.sh “yum install -y cloudera-manager-daemons.x86_64” all
./cmd.sh “yum install -y bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt” all
./cmd.sh “yum install -y cloudera-manager-agent.x86_64” all
安装cloudera-manager-server
yum install -y cloudera-manager-server

配置cm-agent
vim /etc/cloudera-scm-agent/config.ini
server_host=IT-ES-Node01
分发： ./dep.sh /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/ all

修改server的db.properties
vim /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=IT-ES-Node01:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=root
com.cloudera.cmf.db.password=TestCDH2024
com.cloudera.cmf.db.setupType=EXTERNAL

初始化数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root TestCDH2024

启动server服务# systemctl start cloudera-scm-server
查看server状态# systemctl status cloudera-scm-server
查看server日志# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

启动agent服务# ./cmd.sh “systemctl start cloudera-scm-agent” all
查看agent状态# ./cmd.sh “systemctl status cloudera-scm-agent” all
查看agent日志# tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

默认安装目录：
ll /opt/cloudera/
drwxr-xr-x 27 cloudera-scm cloudera-scm 8192 Sep 20 16:03 cm
drwxr-xr-x 8 root root 128 Sep 20 16:10 cm-agent
drwxr-xr-x 2 cloudera-scm cloudera-scm 6 Sep 25 2019 csd
drwxr-xr-x 2 cloudera-scm cloudera-scm 6 Sep 25 2019 parcel-repo

集群安装
http://10.11.17.38:7180/cmf/login amdin admin

分发安装parcel不执行！
cm agent主机异常Error, CM server guid updated, expected
产生的原因是服务器以前作为集群节点安装过agent服务，再次使用时没有卸载干净。
解决办法：
rm -rf /var/lib/cloudera-scm-agent/cm_guid
service cloudera-scm-agent restart

安装完后告警：CDH 以下网络接口似乎未全速运行，virbr-nic。 4主机接口网络似乎为全速运行，cloudera manager agent 无法确定双工模式或接口速度
解决办法：
机器的配置中搜索：网络
修改网络接口收集排除正则表达式为： ^virbr

查看全文

http://www.kler.cn/a/321103.html