启动有几个部分:
1,hdfs;
其中包括、namenode、datanode、secondaryNamenode、backupNode
在namenode上执行:./start-dfs.sh 即可启动nn、dn、snn,
backupNode需要在backupNode上去执行:
nohup ./hdfs namenode -backup > backupNode.out &
2,yarn的启动:
直接./start-yarn.sh即可,没啥难的。
这个脚本自动会将resourcemanager和 resourceNode启动。
然后就可以准备跑第一MapReduce程序啦。
但是在之前,需要建立mr的配置文件,
我不知道为什么,apache的0.23.1的HADOOP_HOME/etc/hadoop里面没有mapred-site.xml,
需要建立,然后对其进行配置,如果不配置会在本地跑mr程序。
我配置了以下几项:
<property>
<description>Execution framework set to Hadoop YARN.</description>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<description>Larger resource limit for maps.</description>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<description>Larger heap-size for child jvms of maps.</description>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<description>Larger resource limit for reduces.</description>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<description>Larger heap-size for child jvms of reduces.</description>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2560M</value>
</property>
<property>
<description>Higher memory-limit while sorting data for efficiency.</description>
<name>mapreduce.task.io.sort.mb</name>
<value>512</value>
</property>
<property>
<description>More streams merged at once while sorting files.</description>
<name>mapreduce.task.io.sort.factor</name>
<value>100</value>
</property>
<property>
<description>
Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.
</description>
<name>mapreduce.reduce.shuffle.parallelcopies</name>
<value>50</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>rmHost:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>rmHost:19888</value>
</property>
<property>
<description>More streams merged at once while sorting files.</description>
<name>mapreduce.task.io.sort.factor</name>
<value>100</value>
</property>
然后就可以跑mr程序啦。
执行例子程序:
hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.1.jar wordcount input output
.
另外关于web查看的入口:
Daemon Web Interface Notes
NameNode http://nn_host:port/ Default HTTP port is 50070.
ResourceManager http://rm_host:port/ Default HTTP port is 8088.
MapReduce JobHistory Server http://jhs_host:port/ Default HTTP port is 19888.
参考资料:
http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html