当前位置： 首页 > article >正文

Hadoop大数据应用：HDFS 集群节点缩容

article 2025/1/19 14:17:22

目录

一、实验

1.环境

2.HDFS 集群节点缩容

二、问题

1.数据迁移有哪些状态

2.数据迁移失败

一、实验

1.环境

（1）主机

表1  主机

主机架构软件版本IP备注
hadoop

NameNode （已部署）

SecondaryNameNode （已部署）

ResourceManager（已部署）

hadoop

2.7.7192.168.204.50

node01

DataNode（已部署）

NodeManager（已部署）

hadoop

2.7.7192.168.204.51
node02

DataNode（已部署）

NodeManager（已部署）

hadoop

2.7.7192.168.204.52
node03

DataNode（已部署）

NodeManager（已部署）

hadoop

2.7.7192.168.204.53
node04

DataNode（已部署）

hadoop

2.7.7192.168.204.54需求迁移主机

（2）查看jps

hadoop节点

[root@hadoop ~]# jps

node01节点

node02节点

node03节点

node04节点

2.HDFS 集群节点缩容

（1）查看节点状态 （已使用）

node01: 176KB (约等于0.18M )

node02: 280KB  (约等于0.28M )

node03: 260KB  (约等于0.26M )

node04:  8KB （约等于0.008M）

[root@hadoop hadoop]# ./bin/hdfs dfsadmin -report
Configured Capacity: 822126559232 (765.67 GB)
Present Capacity: 798788423680 (743.93 GB)
DFS Remaining: 798787682304 (743.93 GB)
DFS Used: 741376 (724 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 5658730496 (5.27 GB)
DFS Remaining: 199872901120 (186.15 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.25%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:33 CST 2024


Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 266240 (260 KB)
Non DFS Used: 5620936704 (5.23 GB)
DFS Remaining: 199910436864 (186.18 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.27%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:33 CST 2024


Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 180224 (176 KB)
Non DFS Used: 6029176832 (5.62 GB)
DFS Remaining: 199502282752 (185.80 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:34 CST 2024


Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 286720 (280 KB)
Non DFS Used: 6029291520 (5.62 GB)
DFS Remaining: 199502061568 (185.80 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:34 CST 2024

（2）查看

[root@hadoop ~]# cd /usr/local/hadoop/
[root@hadoop hadoop]# ./bin/hadoop fs -ls /

（3）上传第一个测试文件

[root@hadoop ~]# /usr/local/hadoop/bin/hadoop fs -put /root/jenkins.tar.gz /jenkins.tar.gz

观察上传变化

[root@hadoop hadoop]# ./bin/hadoop fs -ls /

（4）上传第二个测试文件

[root@hadoop ~]# /usr/local/hadoop/bin/hadoop fs -put /root/pinpoint-image.tar.gz /pinpoint.tar.gz

 观察上传变化

[root@hadoop hadoop]# ./bin/hadoop fs -ls /

（5）web查看

http://192.168.204.50:50070/

（6）清空本地文件

[root@hadoop ~]# rm -rf jenkins.tar.gz pinpoint-image.tar.gz

（7）再次查看节点状态 （已使用）

node01: 2.41GB 

node02: 2.27GB 

node03: 2.15GB  

node04:  2.26GB 

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 822126559232 (765.67 GB)
Present Capacity: 798784053248 (743.93 GB)
DFS Remaining: 789019389952 (734.83 GB)
DFS Used: 9764663296 (9.09 GB)
DFS Used%: 1.22%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5662638080 (5.27 GB)
DFS Remaining: 197444509696 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:57 CST 2024


Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2312085504 (2.15 GB)
Non DFS Used: 5620936704 (5.23 GB)
DFS Remaining: 197598617600 (184.03 GB)
DFS Used%: 1.12%
DFS Remaining%: 96.14%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:58 CST 2024


Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2592927744 (2.41 GB)
Non DFS Used: 6029148160 (5.62 GB)
DFS Remaining: 196909563904 (183.39 GB)
DFS Used%: 1.26%
DFS Remaining%: 95.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:57 CST 2024


Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2435158016 (2.27 GB)
Non DFS Used: 6029783040 (5.62 GB)
DFS Remaining: 197066698752 (183.53 GB)
DFS Used%: 1.18%
DFS Remaining%: 95.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:59 CST 2024

（8）计算总存储使用量

9.09 G （jenkins 1.24 GB,pinpoint 3.27 GB ,存储副本量为2）

(9)修改配置文件（hadoop节点）

#添加代码

[root@hadoop hadoop]# vim hdfs-site.xml

修改前：

修改后：

……
      <property>
          <name>dfs.hosts.exclude</name>
          <value>/usr/local/hadoop/etc/hadoop/exclude</value>
      </property>
……

（10）配置移除主机

[root@hadoop hadoop]# vim /usr/local/hadoop/etc/hadoop/exclude


node04

（11）数据刷新

[root@hadoop hadoop]# ./bin/hdfs dfsadmin -refreshNodes

（12）查看节点状态 

node04节点状态变为Decommissioned in Program

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 600866701319 (559.60 GB)
DFS Remaining: 590966771712 (550.38 GB)
DFS Used: 9899929607 (9.22 GB)
DFS Used%: 1.65%
Under replicated blocks: 2
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommission in progress
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663162368 (5.27 GB)
DFS Remaining: 197443985408 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:34:33 CST 2024


Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2312085504 (2.15 GB)
Non DFS Used: 5756112896 (5.36 GB)
DFS Remaining: 197463441408 (183.90 GB)
DFS Used%: 1.12%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Thu Mar 14 16:34:31 CST 2024


Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2592927744 (2.41 GB)
Non DFS Used: 6232051712 (5.80 GB)
DFS Remaining: 196706660352 (183.20 GB)
DFS Used%: 1.26%
DFS Remaining%: 95.71%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 3
Last contact: Thu Mar 14 16:34:33 CST 2024


Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2570424327 (2.39 GB)
Non DFS Used: 6164545529 (5.74 GB)
DFS Remaining: 196796669952 (183.28 GB)
DFS Used%: 1.25%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Thu Mar 14 16:34:32 CST 2024


Decommissioning datanodes (1):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommission in progress
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663162368 (5.27 GB)
DFS Remaining: 197443985408 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:34:33 CST 2024

(13) 再次查看节点

node04节点状态变为Decommissioned

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 601338761216 (560.04 GB)
DFS Remaining: 589149618176 (548.69 GB)
DFS Used: 12189143040 (11.35 GB)
DFS Used%: 2.03%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommissioned
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5662666752 (5.27 GB)
DFS Remaining: 197444481024 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:33 CST 2024


Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3123707904 (2.91 GB)
Non DFS Used: 5620969472 (5.23 GB)
DFS Remaining: 196786962432 (183.27 GB)
DFS Used%: 1.52%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:31 CST 2024


Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2998738944 (2.79 GB)
Non DFS Used: 6029717504 (5.62 GB)
DFS Remaining: 196503183360 (183.01 GB)
DFS Used%: 1.46%
DFS Remaining%: 95.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:33 CST 2024


Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3642204160 (3.39 GB)
Non DFS Used: 6029963264 (5.62 GB)
DFS Remaining: 195859472384 (182.41 GB)
DFS Used%: 1.77%
DFS Remaining%: 95.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:32 CST 2024

(14) 重启

[root@hadoop hadoop]# ssh node04

[root@node04 ~]# reboot

（15）查看节点（已使用）

node01: 2.79GB 

node02: 3.39GB 

node03: 2.91GB  

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 601338118144 (560.04 GB)
DFS Remaining: 589148975104 (548.69 GB)
DFS Used: 12189143040 (11.35 GB)
DFS Used%: 2.03%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3123707904 (2.91 GB)
Non DFS Used: 5621551104 (5.24 GB)
DFS Remaining: 196786380800 (183.27 GB)
DFS Used%: 1.52%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:14 CST 2024


Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2998738944 (2.79 GB)
Non DFS Used: 6029742080 (5.62 GB)
DFS Remaining: 196503158784 (183.01 GB)
DFS Used%: 1.46%
DFS Remaining%: 95.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:12 CST 2024


Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3642204160 (3.39 GB)
Non DFS Used: 6030000128 (5.62 GB)
DFS Remaining: 195859435520 (182.41 GB)
DFS Used%: 1.77%
DFS Remaining%: 95.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:12 CST 2024


Dead datanodes (1):

Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommissioned
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663240192 (5.27 GB)
DFS Remaining: 197443907584 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:47:27 CST 2024

计算总使用量

（16）回顾使用量变化

表2  数据迁移

主机初始上传迁移前迁移后IP备注
node0176KB 

jenkins 1.24 GB

pinpoint 3.27 GB 

2.41GB2.79GB 192.168.204.51

replication 2

(副本备份量)

node02280KB  2.27GB 3.39GB 192.168.204.52
node03260KB  2.15GB  2.91GB  192.168.204.53
node048KB 2.26GB /192.168.204.54迁移主机
总大小9.09GB9.09GB

二、问题

1.数据迁移有哪些状态

（1）状态

1）正常状态
Normal

2)数据正在迁移
Decommissioned in Progress

3)数据迁移完成 （仅变成当前状态才能down机下线）
Decommissioned

2.数据迁移失败

（1）失败

迁移node04状态依然为Normal

（2）原因分析

配置错误。

（3）解决方法

修改配置。

修改前：

修改后：

成功：


http://www.kler.cn/a/272687.html

相关文章：

  • 放大芯片参数阅读
  • 对话 TDengine 解决方案中心总经理陈肃：构建技术与市场的桥梁
  • 在.NET用C#将Word文档转换为HTML格式
  • turtle教学课程课堂学习考试在线网站
  • 26个开源Agent开发框架调研总结（一）
  • 群论学习笔记
  • Windows系统安装GeoServe结合内网穿透实现公网访问本地位置信息服务
  • C语言学习笔记day8
  • 1057：简单计算器
  • onnx 格式模型可视化工具
  • 疫情网课管理系统|基于springboot框架+ Mysql+Java+Tomcat的疫情网课管理系统设计与实现(可运行源码+数据库+设计文档+部署说明)
  • 网络安全实训Day5
  • 开源模型应用落地-qwen模型小试-合并Lora模型-进阶篇（八）
  • 比特币,区块链及相关概念简介(一)
  • Oracle中的commit与rollback
  • 27-Java MVC 模式
  • WebRTC实现一对多直播模式和弹幕发送功能
  • 【机器学习】无监督学习：解锁数据中的潜在结构与关系
  • rsa数据加密无大小限制——golang实现
  • 华为认证大数据工程师（HCIA-Big Data）--填空题
  • 回到街头 - 数字时尚嘉年华：Web3的时尚未来，4月香港兰桂坊盛大启幕
  • SSM框架，MyBatis-Plus的学习（下）
  • 代码+视频，R语言使用BOOT重抽样获取cox回归方程C-index（C指数）可信区间
  • 闯关升级游戏特点,闯关小程序游戏开发
  • acwing算法提高之搜索--剪枝
  • Verilog中`include的用法