当前位置：首页 > article >正文

本地YARN集群部署

article 2025/3/9 21:05:05

请先完成HDFS的前置部署，部署方式可查看:本地部署HDFS集群https://blog.csdn.net/m0_73641796/article/details/145998092?spm=1001.2014.3001.5502

部署说明

组件	配置文件	启动进程	备注
Hadoop HDFS	需修改	需启动: NameNode作为主节点 DataNode作为从节点 SecondaryNameNode主节点辅助	分布式文件系统
Hadoop YARN	需修改	需启动 ResourceManager作为集群资源管理者 NodeManager作为单机资源管理者 ProxyServer代理服务器提供安全性 JobHistoryServer记录历史信息和日志	分布式资源调度
Hadoop MapReduce	需修改	无需启动任何进程 MapReduce程序运行在YARN容器内	分布式数据计算

集群规划

主机	角色
node1	ResourceManager NodeManager ProxyServer JobHistoryServer
node2	NodeManager
node3	NodeManager

修改MapReduce配置文件

在$HADOOP_HOME/etc/hadoop文件夹内，修改mapred-env.sh文件，添加如下环境变量

export JAVA_HOME=/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

修改mapred-site.xml文件，添加如下配置信息

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>node1:10020</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>node1:19888</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>/data/mr-history/tmp</value>
    <description></description>
  </property>
  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>/data/mr-history/done</value>
    <description></description>
  </property>
 <property>
   <name>yarn.app.mapreduce.am.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
 <property>
   <name>mapreduce.map.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
 <property>
   <name>mapreduce.reduce.env</name>
   <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
 </property>
</configuration>

修改YARN配置文件

在$HADOOP_HOME/etc/hadoop文件夹内，修改yarn-env.sh文件，添加如下环境变量

export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs

修改yarn-site.xml文件，添加如下内容：

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://node1:19888/jobhistory/logs</value>
    <description></description>
</property>
  <property>
    <name>yarn.web-proxy.address</name>
    <value>node1:8089</value>
    <description>proxy server hostname and port</description>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
    <description>Configuration to enable or disable log aggregation</description>
  </property>
  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
    <description>Configuration to enable or disable log aggregation</description>
  </property>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node1</value>
    <description></description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description></description>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data/nm-local</value>
    <description>Comma-separated list of paths on the local filesystem where intermediate data is written.</description>
  </property>
  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data/nm-log</value>
    <description>Comma-separated list of paths on the local filesystem where logs are written.</description>
  </property>
  <property>
    <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
    <description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.</description>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <description>Shuffle service that needs to be set for Map Reduce applications.</description>
  </property>
</configuration>

分发配置文件

scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node2:`pwd`/
scp mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml node3:`pwd`/

启动HDFS

start-dfs.sh

启动YARN

# 一键启动YARN集群
start-yarn.sh

# 一键停止YARN集群
stop-yarn.sh

# 在当前机器，单独启动或停止进程
yarn --daemon start|stop resourcemanager|nodemanager|proxyserver

# 历史服务器启动和停止
mapred --daemon start|stop historyserver

可通过jps命令查看当前运行的进程