从零开始搭建个人大数据集群——环境准备篇
从零开始搭建个人大数据集群（1）——zookeeper
从零开始搭建个人大数据集群（2）——HDFS

本次安装是基于我的上一篇文章——HDFS篇
yarn本身就是hadoop中集成的一个组件，所以这里只需要修改yarn的配置文件就行了

配置说明

本次对yarn的配置是沿用HDFS的配置，将hd1,hd2作为ResourceManager，hd3,hd4,hd5作为NodeManager，同时令hd1为JobHistoryServer

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
        <!-- 重要开始 start -->
        <property>
                <name>yarn.scheduler.minimum-allocation-vcores</name>
                <value>1</value>
                <description>单个任务可申请的最小虚拟CPU个数</description>
        </property>

        <property>
                <name>yarn.scheduler.maximum-allocation-vcores</name>
                <value>3</value>
                <description>单个任务可申请的最大虚拟CPU个数，此参数对应yarn.nodemanager.resource.cpu-vcores，建议最大为一个物理CPU的数量</description>
        </property>

        <property>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>1024</value>
                <description>该节点上可分配的物理内存总量</description>
        </property>

        <property>
                <name>yarn.nodemanager.resource.cpu-vcores</name>
                <value>3</value>
                <description>该节点上YARN可使用的虚拟CPU个数，一个物理CPU对应3个虚拟CPU</description>
        </property>

        <property>
                <name>yarn.scheduler.maximum-allocation-mb</name>
                <value>43008</value>
                <description>单个任务可申请的最多物理内存量</description>
        </property>
        <!-- 重要开始 end -->

        <!-- yarn ha start -->
        <property>
                <name>yarn.resourcemanager.ha.enabled</name>
                <value>true</value>
                <description>是否开启yarn ha</description>
        </property>

        <property>
                <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
                <value>true</value>
                <description>ha状态切换为自动切换</description>
        </property>

        <property>
                <name>yarn.resourcemanager.ha.rm-ids</name>
                <value>rm1,rm2</value>
                <description>RMs的逻辑id列表</description>
        </property>
        <property>
                <name>yarn.resourcemanager.hostname.rm1</name>
                <value>hd1</value>
         </property>

        <property>
                <name>yarn.resourcemanager.hostname.rm2</name>
                <value>hd2</value>
        </property>
        <property>
                <name>yarn.resourcemanager.zk-address</name>
                <value>hd1:2181,hd2:2181,hd3:2181</value>
                <description>ha状态的存储地址</description>
        </property>
        <!-- yarn ha end -->

        <!-- RM1 configs start -->
        <property>
                <name>yarn.resourcemanager.address.rm1</name>
                <value>hd1:8032</value>
                <description>ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等</description>
        </property>

        <property>
                <name>yarn.resourcemanager.scheduler.address.rm1</name>
                <value>hd1:8030</value>
                <description>ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资>源等。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.https.address.rm1</name>
                <value>hd1:8089</value>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.address.rm1</name>
                <value>hd1:8088</value>
                <description>ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
                <value>hd1:8031</value>
                <description>ResourceManager 对NodeManager暴露的地址.。NodeManager通过该地址向RM汇报心跳，领取任务等。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.admin.address.rm1</name>
                <value>hd1:8033</value>
                <description>ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等</description>
        </property>
        <!-- RM1 configs end -->

        <!-- RM2 configs start -->
        <property>
                <name>yarn.resourcemanager.address.rm2</name>
                <value>hd2:8032</value>
                <description>ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等</description>
        </property>

        <property>
                <name>yarn.resourcemanager.scheduler.address.rm2</name>
                <value>hd2:8030</value>
                <description>ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资>源等。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.https.address.rm2</name>
                <value>hd2:8089</value>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.address.rm2</name>
                <value>hd2:8088</value>
                <description>ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
                <value>hd2:8031</value>
                <description>ResourceManager 对NodeManager暴露的地址.。NodeManager通过该地址向RM汇报心跳，领取任务等。</description>
        </property>

        <property>
                <name>yarn.resourcemanager.admin.address.rm2</name>
                <value>hd2:8033</value>
                <description>ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等</description>
        </property>
        <!-- RM2 configs end -->

        <!-- scheduler begin -->
        <property>
                <name>yarn.resourcemanager.scheduler.class</name>
                <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
                <description>调度器实现类</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.allocation.file</name>
                <value>fair-scheduler.xml</value>
                <description>自定义XML配置文件所在位置，该文件主要用于描述各个队列的属性，比如资源量、权重等</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.user-as-default-queue</name>
                <value>true</value>
                <description>当应用程序未指定队列名时，是否指定用户名作为应用程序所在的队列名。如果设置为false或者未设置，所有未知队列的应用程序将被提交到default队列中，默认值为true</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.preemption</name>
                <value>true</value>
                <description>是否支持抢占</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.sizebasedweight</name>
                <value>false</value>
                <description>在一个队列内部分配资源时，默认情况下，采用公平轮询的方法将资源分配各各个应用程序，而该参数则提供了外一种资源分配方式：按照应用程序资源需求数目分配资源，即需求资源数量越多，分配的资源越多。默认情况下，该参数值为false</description>
        </property>

        <property>
                <name>yarn.scheduler.increment-allocation-mb</name>
                <value>256</value>
                <description>内存规整化单位，默认是1024，这意味着，如果一个Container请求资源是700mB，则将被调度器规整化为 (700mB / 256mb) *  256mb=768mb</description>
        </property>

        <property>
                <name>yarn.scheduler.assignmultiple</name>
                <value>true</value>
                <description>是否启动批量分配功能。当一个节点出现大量资源时，可以一次分配完成，也可以多次分配完成。默认情况下，参数值为false</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.max.assign</name>
                <value>10</value>
                <description>如果开启批量分配功能，可指定一次分配的container数目。默认情况下，该参数值为-1，表示不限制</description>
        </property>

        <property>
                <name>yarn.scheduler.fair.allow-undeclared-pools</name>
                <value>false</value>
                <description>如果提交的队列名不存在，Scheduler会自动创建一个该队列，默认开启</description>
        </property>
        <!-- scheduler end -->


        <!--目录相关 start -->
        <property>
                <name>yarn.nodemanager.local-dirs</name>
                <value>/data/yarn/local</value>
                <description>中间结果存放位置，存放执行Container所需的数据如可执行程序或jar包，配置文件等和运行过程中产生的临时数据</description>
        </property>

        <property>
                <name>yarn.nodemanager.log-dirs</name>
                <value>/data/yarn/logs</value>
                <description>Container运行日志存放地址（可配置多个目录）</description>
        </property>

        <property>
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
                <description>是否启用日志聚集功能</description>
        </property>
        <!-- 配置日志服务器的地址,work节点使用 -->
        <property>
                <name>yarn.log.server.url</name>
                <value>hd1:19888/jobhistory/logs/</value>
        </property>
        <property>
                <name>yarn.nodemanager.remote-app-log-dir</name>
                <value>/tmp/app-logs</value>
                <description>当应用程序运行结束后，日志被转移到的HDFS目录（启用日志聚集功能时有效）</description>
        </property>
        <!--目录相关 end -->

        <!-- 其它 -->
        <property>
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>1209600</value>
                <description>nodemanager上所有Container的运行日志在HDFS中的保存时间，保留半个月</description>
        </property>

        <property>
                <name>yarn.nodemanager.address</name>
                <value>0.0.0.0:9103</value>
        </property>

        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
                <description>NodeManager上运行的附属服务。需配置成mapreduce_shuffle，才可运行MapReduce程序</description>
        </property>

        <property>
                <name>yarn.resourcemanager.cluster-id</name>
                <value>yarn-cluster</value>
                <description>集群的Id</description>
        </property>

        <property>
                <name>yarn.resourcemanager.recovery.enabled</name>
                <value>true</value>
                <description>默认值为false，也就是说resourcemanager挂了相应的正在运行的任务在rm恢复后不能重新启动</description>
        </property>

        <property>
                <name>yarn.resourcemanager.store.class</name>
                <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
                <description>配置RM状态信息存储方式3有两种，一种是FileSystemRMStateStore,另一种是MemoryRMStateStore，还有一种目前较为主流的是zkstore</description>
        </property>

        <property>
                <name>yarn.resourcemanager.zk.state-store.address</name>
                <value>hd1:2181,hd2:2181,hd3:2181</value>
                <description>当使用ZK存储时，指定在ZK上的存储地址。</description>
        </property>

        <property>
                <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
                <value>5000</value>
        </property>

        <property>
                <name>yarn.nodemanager.webapp.address</name>
                <value>0.0.0.0:8042</value>
        </property>

        <property>
                <name>yarn.nodemanager.localizer.address</name>
                <value>0.0.0.0:8040</value>
        </property>

        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>

        <property>
                <name>mapreduce.shuffle.port</name>
                <value>23080</value>
        </property>

        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user</value>
        </property>

        <property>
                <name>yarn.web-proxy.address</name>
                <value>hd1:8041</value>
        </property>

        <property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
                <description>虚拟内存检测，默认是True</description>
        </property>

        <property>
                <name>yarn.nodemanager.pmem-check-enabled</name>
                <value>false</value>
                <description>物理内存检测，默认是True</description>
        </property>
        <property>
                <name>yarn.application.classpath</name>
                <value>/opt/apps/hadoop/etc/hadoop:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/hdfs:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/hdfs/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/yarn:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*</value>
        </property>

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 指定mr框架 -->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<!-- 这一段必须写，不然yarn执行任务时会报
     Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster -->
<property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<!-- JobHistoryServer相关配置 -->
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hd1:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hd1:19888</value>
</property>
<property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>/user/history/done</value>
</property>

<property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>/user/history/done_intermediate</value>
</property>
<!-- MR运行相关配置 -->
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>1000</value>
        <description>表示MRAppMaster需要的总内存大小，默认是1536</description>
</property>
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx800m</value>
        <description>表示MRAppMaster需要的堆内存大小，默认是：-Xmx1024m</description>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
    <value>1</value>
    <description>表示MRAppMaster需要的的虚拟cpu数量，默认是：1</description>
</property>

<property>
    <name>mapreduce.map.memory.mb</name>
    <value>512</value>
        <description>表示MapTask需要的总内存大小，默认是1024</description>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx300m</value>
    <description>表示MapTask需要的堆内存大小，默认是-Xmx200m</description>
</property>
<property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
    <description>表示MapTask需要的虚拟cpu大小，默认是1</description>
</property>

<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>512</value>
    <description>表示ReduceTask需要的总内存大小，默认是1024</description>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx300m</value>
    <description>表示ReduceTask需要的堆内存大小，默认是-Xmx200m</description>
</property>
<property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
    <description>表示ReduceTask需要的虚拟cpu大小，默认是1</description>
</property>

</configuration>

配置完成后参考我这篇文章完成脚本
Linux多机文件批量同步脚本
用该脚本将上述两个配置文件同步到所有的节点

启动

1.启动zookeeper
~/zk/ssh_all.sh zkServer.sh start
2.启动HDFS及YARN
start-all.sh
这里也可以分别启动
start-dfs.sh start-yarn.sh
3.在hd1上启动yarn日志服务
mr-jobhistory-daemon.sh start historyserver

Q.E.D.