在Centos7.9_2207安装CDH6.3.2

在Centos7.9_2207安装CDH6.3.2

背景

笔者做大数据开发，实时部分一般要用到HBase、Kudu、Redis等组件来保证幂等性，为了方便，还是选用老古董CDH6.3.2【最后的免费版】做一个单节点机器，方便随时挂起。多节点虚拟机由之前的双路E5做Frp后远程调用。

物理机

资源配置

机型：惠普战99锐龙版

CPU：7840HS 8c16t

RAM：2*48G

SSD：1T+4T

很普通的集显工作站。2条DDR5的48G内存条组成96G内存。

部署好CDH常用组件后，目前的资源占用情况大概是这样：

经过多年的发展，轻薄本用的7840HS已经远远超越了2016年的标压6700hq，如果没有便携或者大内存的需求，上个7945HX，可以和双路E5-2696V3打的有来有回。

安装Centos7.9_2207

下载ISO镜像

https://wiki.centos.org/Download.html
http://isoredirect.centos.org/centos/7/isos/x86_64/
https://mirrors.tuna.tsinghua.edu.cn/centos/7.9.2009/isos/x86_64/

下载CDH安装包：

https://archive.cloudera.com/

资源配置

CPU：4c

RAM：48G

硬盘：300G

因为要搭一个单节点CDH，所有组件会全部在一个虚拟机启动，所以尽可能分配充足的资源。

网络配置

IP：192.168.88.52

子网掩码：255.255.255.0

网关：192.168.88.2

DNS：8.8.8.8

host：bigdata-

禁用IPV6。

此时：

[root@bigdata-centos ~]# hostname
bigdata-centos
[root@bigdata-centos ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@bigdata-centos ~]#

部署虚拟机

zhiyong
Shyf@2019

正常部署即可。

CPU性能测试

在CentOS上安装UnixBench可以按照以下步骤进行：

1. 打开终端，使用root用户或具有sudo权限的用户登录。

2. 确保您的系统已安装了必要的依赖关系。运行以下命令来安装这些依赖关系：

sudo yum install make gcc
mkdir -p /home/bigdata/unixbench
cd /home/bigdata/unixbench
wget https://codeload.github.com/kdlucas/byte-unixbench/zip/refs/tags/v5.1.3
unzip byte-unixbench-5.1.3.zip
cd /home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench
make ./Run

耐心等待跑完后，可以看到跑分情况，能凑合用：

[root@bigdata-centos UnixBench]# ./Run
make all
make[1]: Entering directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
Checking distribution of files
./pgms  exists
./src  exists
./testdir  exists
./tmp  exists
./results  exists
make[1]: Leaving directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
sh: 3dinfo: command not found

   #    #  #    #  #  #    #          #####   ######  #    #   ####   #    #
   #    #  ##   #  #   #  #           #    #  #       ##   #  #    #  #    #
   #    #  # #  #  #    ##            #####   #####   # #  #  #       ######
   #    #  #  # #  #    ##            #    #  #       #  # #  #       #    #
   #    #  #   ##  #   #  #           #    #  #       #   ##  #    #  #    #
    ####   #    #  #  #    #          #####   ######  #    #   ####   #    #

   Version 5.1.3                      Based on the Byte Magazine Unix Benchmark

   Multi-CPU version                  Version 5 revisions by Ian Smith,
                                      Sunnyvale, CA, USA
   January 13, 2011                   johantheghost at yahoo period com


1 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

1 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

1 x Execl Throughput  1 2 3

1 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

1 x File Copy 256 bufsize 500 maxblocks  1 2 3

1 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

1 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

1 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

1 x Process Creation  1 2 3

1 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

1 x Shell Scripts (1 concurrent)  1 2 3

1 x Shell Scripts (8 concurrent)  1 2 3

4 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

4 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

4 x Execl Throughput  1 2 3

4 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

4 x File Copy 256 bufsize 500 maxblocks  1 2 3

4 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

4 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

4 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

4 x Process Creation  1 2 3

4 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

4 x Shell Scripts (1 concurrent)  1 2 3

4 x Shell Scripts (8 concurrent)  1 2 3

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: bigdata-centos: GNU/Linux
   OS: GNU/Linux -- 3.10.0-1160.71.1.el7.x86_64 -- #1 SMP Tue Jun 28 15:37:28 UTC 2022
   Machine: x86_64 (x86_64)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 1: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 2: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 3: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:36:48 up 43 min,  3 users,  load average: 0.00, 0.01, 0.05; runlevel 5

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 01:36:48 - 02:05:51
4 CPUs in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       45522028.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3556.8 MWIPS (15.8 s, 7 samples)
Execl Throughput                               3186.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        792934.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          215891.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2162307.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1915569.4 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  20637.0 lps   (10.0 s, 7 samples)
Process Creation                               8614.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7681.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   2594.1 lpm   (60.0 s, 2 samples)
System Call Overhead                        2884075.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   45522028.9   3900.8
Double-Precision Whetstone                       55.0       3556.8    646.7
Execl Throughput                                 43.0       3186.7    741.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     792934.6   2002.4
File Copy 256 bufsize 500 maxblocks            1655.0     215891.9   1304.5
File Copy 4096 bufsize 8000 maxblocks          5800.0    2162307.5   3728.1
Pipe Throughput                               12440.0    1915569.4   1539.8
Pipe-based Context Switching                   4000.0      20637.0     51.6
Process Creation                                126.0       8614.4    683.7
Shell Scripts (1 concurrent)                     42.4       7681.2   1811.6
Shell Scripts (8 concurrent)                      6.0       2594.1   4323.4
System Call Overhead                          15000.0    2884075.7   1922.7
                                                                   ========
System Benchmarks Index Score                                        1252.4

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 02:05:51 - 02:34:44
4 CPUs in system; running 4 parallel copies of tests

Dhrystone 2 using register variables      151547416.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                    14886.9 MWIPS (13.9 s, 7 samples)
Execl Throughput                              15018.6 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        829648.1 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          217171.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2729918.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                             6317809.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 846011.1 lps   (10.0 s, 7 samples)
Process Creation                              54076.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                  20820.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   3192.6 lpm   (60.1 s, 2 samples)
System Call Overhead                        8667731.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0  151547416.3  12986.1
Double-Precision Whetstone                       55.0      14886.9   2706.7
Execl Throughput                                 43.0      15018.6   3492.7
File Copy 1024 bufsize 2000 maxblocks          3960.0     829648.1   2095.1
File Copy 256 bufsize 500 maxblocks            1655.0     217171.6   1312.2
File Copy 4096 bufsize 8000 maxblocks          5800.0    2729918.6   4706.8
Pipe Throughput                               12440.0    6317809.1   5078.6
Pipe-based Context Switching                   4000.0     846011.1   2115.0
Process Creation                                126.0      54076.5   4291.8
Shell Scripts (1 concurrent)                     42.4      20820.9   4910.6
Shell Scripts (8 concurrent)                      6.0       3192.6   5321.0
System Call Overhead                          15000.0    8667731.7   5778.5
                                                                   ========
System Benchmarks Index Score                                        3863.4

[root@bigdata-centos UnixBench]#

准备工作

配置SSH免密

要配置CentOS上的免密登录并将密钥发送到其他机器，可以按照以下步骤进行：

1. 在bigdata-centos机器上生成SSH密钥对。打开终端并运行以下命令：

[root@bigdata-centos ~]# ssh-keygen -t rsa	#按3次回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:MLaVLFiyOmRjJ3awp9Kb3kh2Y53wSf8P3hUojvZyeho root@bigdata-centos
The key's randomart image is:
+---[RSA 2048]----+
|  . . .          |
|   o = . .       |
|  O * = +        |
| * O . *     .   |
|. = . o S . . .  |
| . + = + o .   . |
|  = + = E o   .  |
| + = . ..=oo .   |
|  o .   o*+.o    |
+----[SHA256]-----+
[root@bigdata-centos ~]# ssh-copy-id root@bigdata-centos
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'bigdata-centos (192.168.88.52)' can't be established.
ECDSA key fingerprint is SHA256:jO6jVOMifwO+jzliWqHgSF2LU39q4UtooXWKEWsVVgc.
ECDSA key fingerprint is MD5:78:5d:df:f1:53:79:f3:73:de:ac:9d:be:dc:c4:3c:40.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@bigdata-centos's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@bigdata-centos'"
and check to make sure that only the key(s) you wanted were added.

[root@bigdata-centos ~]# ssh bigdata-centos
Last login: Wed Oct 18 02:09:26 2023 from 192.168.88.1
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]# exit
登出
Connection to bigdata-centos closed.
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]#

此时SSH免密配置成功。

配置NTP时钟同步

为了使用Kudu，一定要配置好NTP。

[root@bigdata-centos ~]# yum install -y ntp
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
软件包 ntp-4.2.6p5-29.el7.centos.2.x86_64 已安装并且是最新版本
无须任何处理
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# cat /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
#restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

restrict 192.168.88.0 mask 255.255.255.0 nomodify notrap
#server ntp4.aliyun.com
#server 202.112.10.36
server 127.127.1.0 iburst# localclock
fudge 127.127.1.0 stratum 8

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# hwclock --systohc --localtime
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 15:45:54 CST
  Universal time: 四 2023-10-19 07:45:54 UTC
        RTC time: 四 2023-10-19 07:45:54
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos ~]# vim /etc/ntp.conf
您在 /var/spool/mail/root 中有新邮件
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 16:09:23 CST
  Universal time: 四 2023-10-19 08:09:23 UTC
        RTC time: 四 2023-10-19 08:09:23
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos kudu]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*LOCAL(0)        .LOCL.           8 l    2   64    1    0.000    0.000   0.000
[root@bigdata-centos kudu]# rm -rf /home/kududata

主要是Kudu的时钟比较麻烦，所以单节点可以只配置和本机同步。

配置JDK

[root@bigdata-centos ~]# java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)
[root@bigdata-centos ~]#

这个版本的Centos7.9本身自带了JDK1.8，可以直接使用。

关闭透明大页

[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
[root@bigdata-centos ~]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos ~]#

显然当前版本的Centos开启了透明大页功能，需要关闭：

[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# vim /etc/default/grub
[root@bigdata-centos UnixBench]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

关闭后还需要reboot重启生效。

减少swap

[root@bigdata-centos ~]# sysctl -w vm.swappiness=0
vm.swappiness = 0
[root@bigdata-centos ~]# echo 'vm.swappiness=0' >> /etc/sysctl.conf
[root@bigdata-centos ~]# sysctl -p
vm.swappiness = 0
[root@bigdata-centos ~]#

0表示只有当可⽤物理内存⼩于最⼩阈值vm.min_free_kbytes时才使⽤交换空间。

关闭安全配置

[root@bigdata-centos yum.repos.d]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@bigdata-centos yum.repos.d]# systemctl stop firewalld
[root@bigdata-centos yum.repos.d]# vim /etc/selinux/config
[root@bigdata-centos yum.repos.d]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disable
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@bigdata-centos yum.repos.d]#

此时关闭了SELinux与防火墙。

安装Http与Repo

yum install -y httpd
yum install -y createrepo
systemctl start httpd
systemctl enable httpd

# Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

此时安装好Http与Repo。浏览器打开：

192.168.88.52:80

看到：

说明Http已经启动。

制作离线Yum源

[root@bigdata-centos yum.repos.d]# mkdir -p /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos yum.repos.d]# vi /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[root@bigdata-centos yum.repos.d]# cat /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[offline]
name=zhiyong-bigdata-centos
baseurl=http://192.168.88.52:80/yum/	# file:///home/bigdata/yum/offline/zhiyong-bigdata-centos/
enabled=1
gpgcheck=0
[root@bigdata-centos repodata]# cd /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos zhiyong-bigdata-centos]# ll
总用量 3599392
-rw-r--r--. 1 root root     277604 10月 18 03:40 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2237116 10月 18 03:40 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2112700 10月 18 03:40 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root   25034716 10月 18 03:40 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root  159295840 10月 18 03:40 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root      14040 10月 18 03:10 allkeys.asc
-rw-r--r--. 1 root root 2082186246 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r--. 1 root root         40 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
-rw-r--r--. 1 root root         64 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
-rw-r--r--. 1 root root   10483568 10月 18 03:10 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root 1203832464 10月 18 03:10 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      11488 10月 18 03:10 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      10996 10月 18 03:10 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root   14209868 10月 18 03:10 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      33887 10月 18 03:10 manifest.json
-rw-r--r--. 1 root root    1006956 10月 18 03:40 mysql-connector-java-5.1.48.jar
-rw-r--r--. 1 root root  184988341 10月 18 03:10 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x. 2 root root       4096 10月 18 03:40 repodata
[root@bigdata-centos zhiyong-bigdata-centos]# createrepo .
[root@bigdata-centos zhiyong-bigdata-centos]# yum clean all
[root@bigdata-centos zhiyong-bigdata-centos]# yum makecache
[root@bigdata-centos zhiyong-bigdata-centos]# mkdir -p /var/www/html/yum
[root@bigdata-centos zhiyong-bigdata-centos]# cp /home/bigdata/yum/offline/zhiyong-bigdata-centos/* /var/www/html/yum

将离线的repo文件使用MobaXterm放置到/home/bigdata/yum/offline/zhiyong-bigdata-centos路径。此时：

http://192.168.88.52/yum/

已经可以看到这些文件。此时可以离线部署【由于Cloudera关闭了官网的下载链接，目前也只能离线部署】。

放置parcel包

/opt/cloudera/parcel-repo
/opt/cloudera/parcels

将CDH组件的4个包在该路径也放置一份。

安装MySQL

[root@bigdata-centos zhiyong-bigdata-centos]# pwd
/home/bigdata/yum/offline/zhiyong-bigdata-centos

rpm -qa|grep mariadb | xargs -I {} rpm -e --nodeps {}
rpm -ivh 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
rpm -ivh 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm

systemctl start mysqld
systemctl enable mysqld

[root@bigdata-centos zhiyong-bigdata-centos]# grep 'temporary password' /var/log/mysqld.log
2023-10-17T19:46:44.706162Z 1 [Note] A temporary password is generated for root@localhost: #DR2-C#qp1wK
[root@bigdata-centos zhiyong-bigdata-centos]# mysql -uroot -p
#DR2-C#qp1wK
alter user root@localhost identified by 'Zhiyong123#';
flush privileges;

[root@bigdata-centos yum]# pwd
/var/www/html/yum
[root@bigdata-centos yum]# cp mysql-connector-java-5.1.48.jar /usr/share/java/
[root@bigdata-centos java]# pwd
/usr/share/java
[root@bigdata-centos java]# mv mysql-connector-java-5.1.48.jar mysql-connector-java.jar

此时MySQL的密码已经修改。

创建元数据库表

mysql -uroot -pZhiyong123# -e 'create database metastore default character set utf8;'
mysql -uroot -pZhiyong123# -e "create user 'hive'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hive default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hive.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "grant all privileges on metastore.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'amon'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database amon default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on amon.* to 'amon'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'rman'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database rman default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on rman.* to 'rman'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'sentry'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database sentry default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on sentry.* to 'sentry'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'nav'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database nav default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on nav.* to 'nav'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'navms'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database navms default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on navms.* to 'navms'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'cm'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database cm default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on cm.* to 'cm'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'oos'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database oos default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on oos.* to 'oos'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'hue'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hue default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hue.* to 'hue'@'%'";

创建成功。此处建库根据需求。

安装CDH

安装CM

yum install cloudera-manager-daemons cloudera-manager-server

初始化CM

[root@bigdata-centos java]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm Zhiyong123#
JAVA_HOME=/usr/lib/jvm/jre-openjdk
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/lib/jvm/jre-openjdk/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Oct 18 04:53:43 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

此时初始化CM的元数据成功。要注意密码不能有!，可以有#。血的教训。

启动CM

systemctl start cloudera-scm-server.service

稍等片刻即可进入熟悉的CM界面。

安装组件

http://192.168.88.52:7180/cmf/login
admin
admin

当然是选择体验版。。。

未在已配置的存储库中找到任何 parcel。尝试在更多选项下添加一个自定义存储库。否则，您可能只能继续使用包。

还需要：

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
systemctl restart cloudera-scm-server

重置Hash以后才可以下一步。

出现无法复制安装文件的情况。。。要取消掉存储库的所有配置项，只保留本地源，这样才能保证不会从已过时的Cloudera官网下载安装包。不然会出现：

/tmp/scm_prepare_node.k8fwJHf3
https://archive.cloudera.com/cm6/6.3.1/allkeys.asc/

类似这种的报错。

此外，由于是单节点，本身是不合理的，所以后续所有检查、告警能跳过的一律跳过。

原则就是能凑合着用即可。

后续操作

允许远程登陆MySQL

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Zhiyong123#' WITH GRANT OPTION;
FLUSH PRIVILEGES;

这样就可以远程登陆MySQL，方便后续查看Hive的元数据。

配置Jdbc连接

找一个Impala的JDBC41驱动包，传入url：

jdbc:impala://192.168.88.52:21050

后续就可以用这种方式访问Impala，间接操作Kudu和Hive，从而忽视Hue不正常的问题。

调度工具Oozie已经要淘汰了，开发环境也没有研究的必要，直接停掉。

FRP端口暴露

使用路由侠，暴露Impala的端口给外部访问。这样就可以让别的只会写Sql的Sql Boy用公网的方式连接到Impala：

jdbc:impala://zhiyong.xxxx这里是具体的网址xxxxxxx.luyouxia.net:11701
jdbc:impala://zhiyong.e3.luyouxia.net:11701

这种方式比租一个ECS要便宜些。延时90ms左右。

此时完成了CDH6.3.2单节点部署的主要操作。

Kudu无法启动问题

出现Kudu无法启动的问题，reboot重置NTP：

[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# ntpstat
synchronised to local net (127.127.1.0) at stratum 11
   time correct to within 7948 ms
   polling server every 64 s

之后：

gflagfile 的 Tablet Server 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
gflagfile 的 Master 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
block_cache_capacity_mb=1G
memory_limit_hard_bytes=8G

即可解决如下报错：

E1019 19:13:24.295385 15847 block_cache.cc:97] Block cache capacity exceeds the memory pressure threshold (8589934592 bytes vs. 5153960755 bytes). This will cause instability and harmful flushing behavior. Lower --block_cache_capacity_mb or raise --memory_limit_hard_bytes.

CM的报错在如下Log查看：

/var/log/cloudera-scm-server

此时可以正常使用。

也可以在Idea中使用JDBC方式访问：

package com.zhiyong.kudu;

import java.sql.*;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.Map;

public class KuduImpalaJdbcDemo {
    public static void main(String[] args) throws ClassNotFoundException, SQLException {
        System.out.println("使用Impala操作Kudu");
        Class.forName("com.cloudera.impala.jdbc41.Driver");
        String url = "jdbc://192.168.88.52:21050";

        Connection connection = DriverManager.getConnection(url);

        ArrayList<Map> result = new ArrayList<>();

        Statement statement = connection.createStatement();
        ResultSet resultSet;
        ResultSetMetaData metaData;
        String sql = "select * from table_name ";

        try {

            resultSet = statement.executeQuery(sql);//结果集
            metaData = resultSet.getMetaData();//元数据集

            while (resultSet.next()){
                LinkedHashMap<Object, Object> map = new LinkedHashMap<>();
                for (int i = 0; i < metaData.getColumnCount(); i++) {
                    map.put(metaData.getColumnName(i+1),resultSet.getObject(metaData.getColumnName(i+1)));
                }
                result.add(map);
            }
        } catch (SQLException e) {
            e.printStackTrace();
        } finally {
            try {
                statement.close();
                connection.close();

            } catch (Exception e){
                e.printStackTrace();
            }
        }

        System.out.println("结束");

    }
}

基本搭建完成。

转载请注明出处：https://lizhiyong.blog.csdn.net/article/details/134433148