当前位置：首页 > article >正文

Hadoop学习笔记（HDP）-Part.04 基础环境配置

article 2024/11/16 13:29:35

目录
Part.01 关于HDP
Part.02 核心组件原理
Part.03 资源规划
Part.04 基础环境配置
Part.05 Yum源配置
Part.06 安装OracleJDK
Part.07 安装MySQL
Part.08 部署Ambari集群
Part.09 安装OpenLDAP
Part.10 创建集群
Part.11 安装Kerberos
Part.12 安装HDFS
Part.13 安装Ranger
Part.14 安装YARN+MR
Part.15 安装HIVE
Part.16 安装HBase
Part.17 安装Spark2
Part.18 安装Flink
Part.19 安装Kafka
Part.20 安装Flume

四、基础环境配置

1.SSH免密登录

在hdp01、hdp02上生成公钥，配置免密登录到其他节点

ssh-keygen -t rsa -f ~/.ssh/id_rsa -C username_root
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.201
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.202
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.203
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.204
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 root@192.168.111.205

2.ansible安装

配置ansible和rhel-system-roles，创建配置文件

mkdir /root/ansible
cd /root/ansible
cp /etc/ansible/ansible.cfg /root/ansible/

修改配置文件，/root/ansible/ansible.cfg

[defaults]
inventory      = /root/ansible/inventory
ask_pass      = false
remote_user = root

配置inventory文件，/root/ansible/inventory

[hdp:children]
nn
dn
[nn]
192.168.111.201 hostname=hdp01
192.168.111.202 hostname=hdp02
[dn]
192.168.111.203 hostname=hdp03
192.168.111.204 hostname=hdp04
192.168.111.205 hostname=hdp05

3.修改hostname

创建playbook，/root/ansible/hostname.yml

---
- name: modify hostname
  hosts: all
  tasks:
    - name: modify hostname permanently
      raw: "echo {{ hostname | quote }} > /etc/hostname"
    - name: modify hostname temporarily
      shell: hostname {{ hostname | quote }}

执行并确认

ansible-playbook /root/ansible/hostname.yml
ansible all -m shell -a 'hostname'

4.修改hosts列表

在nn01上修改主机列表，/etc/hosts

192.168.111.201 hdp01.hdp.com   hdp01
192.168.111.202 hdp02.hdp.com   hdp02
192.168.111.203 hdp03.hdp.com   hdp03
192.168.111.204 hdp04.hdp.com   hdp04
192.168.111.205 hdp05.hdp.com   hdp05

分发至其他节点

ansible all -m template -a 'src=/etc/hosts dest=/etc/hosts'

5.安装基础软件

安装vim等基础软件，/root/ansible/packages.yml

---
- hosts: all
  tasks:
    - name: install packages
      yum:
        name:
          - pciutils
          - bash-completion
          - vim
          - chrony
        state: present

6.关闭firewall和SELinux

关闭firewall

ansible all -m service -a 'name=firewalld state=stopped enabled=no'
ansible all -m shell -a 'systemctl status firewalld | grep Active'

关闭SELinux

ansible all -m selinux -a 'policy=targeted state=disabled'
ansible all -m shell -a 'getenforce'

7.NTP时钟

以hdp01为时钟源，其余节点从nn01进行时钟同步
服务端（hdp01）
修改配置文件/etc/chrony.conf

# 不指定外部NTP源
# 允许本网段其节点作为客户端访问
allow 192.168.111.0/24
# 如果时间服务可不用，则使用本地时间作为标准时间授权，层数为10
local stratum 10

重启服务

systemctl restart chronyd

客户端（hdp02-hdp05）
安装ntp时钟，/root/ansible/timesync.yml

---
- hosts: 192.168.111.202,dn
  vars:
    timesync_ntp_servers:
      - hostname: 192.168.111.201
        iburst: yes
  roles:
    - rhel-system-roles.timesync

执行

ansible-playbook /root/ansible/timesync.yml

确认时钟同步情况

ansible 192.168.111.202,dn -m shell -a 'chronyc sources -v'

8.磁盘分区、文件系统及挂载目录

创建分区parted.yml文件

---
- hosts: all
  tasks:
    - name: parted devices
      parted:
        device: "{{ item }}"
        number: 1
        label: gpt
        state: present
      loop:
        - /dev/sdb
        - /dev/sdc
        - /dev/sdd
      ignore_errors: yes

确认结果

ansible all -m shell -a 'lsblk -f'

创建文件系统mkfs.yml文件

---
- hosts: all
  tasks:
    - name: mkdir of nn
      file:
        path: /data01
        state: directory
      when: inventory_hostname in groups['nn']
    - name: mkdir of dn
      file:
        path: "{{ item }}"
        state: directory
      loop:
        - /data01
        - /data02
        - /data03
      when: inventory_hostname in groups['dn']
    - name: mkfs
      filesystem:
        fstype: xfs
        dev: "{{ item }}"
      loop:
        - /dev/sdb1
        - /dev/sdc1
        - /dev/sdd1
      ignore_errors: yes
    - name: mount of nn
      mount:
        path: /data01
        src: /dev/sdb1
        fstype: xfs
        state: mounted
      when: inventory_hostname in groups['nn']
    - name: mount of dn
      mount:
        path: "{{ item.p_dir }}"
        src: "{{ item.s_dir }}"
        fstype: xfs
        state: mounted
      loop:
        - { p_dir: /data01, s_dir: /dev/sdb1 }
        - { p_dir: /data02, s_dir: /dev/sdc1 }
        - { p_dir: /data03, s_dir: /dev/sdd1 }
      when: inventory_hostname in groups['dn']