Hadoop伪分布式模式部署 HDFS（root用户）_综合

Hadoop伪分布式模式部署

Hadoop2.x：

官网：hadoop.apache.org

三个组件：

HDFS：分布式文件系统，存储

MapReduce：分布式计算

Yarn：资源（cpu + memory）和JOB调度监控

文档地址：

http://hadoop.apache.org/docs/r2.8.2/

部署方式：

1.单机模式（Standalone）：1个Java进程

2.伪分布模式（Pseudo-Distribute Mode）：开发|学习多个Java进程

3.集群模式（Cluster Mode）：生产多态机器多个Java进程

伪分布式部署：HDFS

1.创建hadoop服务的一个用户

# useradd hadoop

# id hadoop

# vi /etc/sudoers

hadoop ALL=(root) NOPASSWD:ALL

2.部署Java

使用：Oracle jdk1.8

尽力不要使用： Open JDk

2.1、解压+环境变量

2.2、CDH课 /usr/java

最好：which java

3.部署ssh，确保其是运行的

查看：默认是已经安装

service sshd status

4.解压Hadoop

解压：

software]# tar -zxvf hadoop-2.8.1.tar.gz
修改权限：

chown -R root:root hadoop-2.8.1

配置环境变量
    vi /etc/profile
    export HADOOP_HOME=/opt/software/hadoop-2.8.1
    export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
    # which hadoop
    # /opt/software/hadoop-2.8.1/bin/hadoop

软连接：

software]# ln -s /opt/software/hadoop-2.8.1 hadoop

修改用户和用户组：

software]# chown -R hadoop:hadoop hadoop

修改软连接

software]# chown -R hadoop:hadoop hadoop/*

修改软连接文件夹里面的内容

software]# chown -R hadoop:hadoop hadoop-2.8.1

修改原文件夹

chown -R hadoop:hadoop 文件夹：修改文件夹和文件夹里面的内容

chown -R hadoop:hadoop 软连接文件夹：修改软连接文件夹，不会修改软连接文件夹里面的

chown -R hadoop:hadoop/* 软连接文件夹/*：软连接文件夹不修改，只修改软连接文件夹里面的

software]# cd hadoop-2.8.1

#删除文件夹内.cmd后缀文件避免混淆

hadoop-2.8.1]# rm -rf /hadoop-2.8.1/bin/*.cmd
hadoop-2.8.1]# rm -rf /hadoop-2.8.1/sbin/*.cmd

bin：执行命令的shell

etc：配置文件

lib：lib库

sbin：启动和关闭hadoop里面的组件（进程）
hadoop-2.8.1/share/hadoop/hdfs：jar包

5.hadoop的模块组件配置core-site.xml

Hadoop-env.sh :hadoop配合环境
core-site.xml :hadoop 核心配置文件
hdfs-site.xml :针对于hdfs服务的会起进程

[mapred-site.xml:MapReduce计算所需要的配置文件] 只有在jar计算时才有

yarn-site.xml :yarn服务的会起进程
slaves :集群的机器名称

配置core-site文件

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

6.配置root用户的ssh的信任关系

~]# rm -rf .ssh

~]# ssh-keygen
~]# cd .ssh
.ssh]# cat id_rsa.pub >> authorized_keys 添加自己的公钥到信任名单

#第一次执行

~]# ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Sun May 13 21:49:14 CST 2018

#第二次执行
~]# ssh localhost date
Sun May 13 21:49:17 CST 2018

7.格式化文件系统

$ bin/hdfs namenode -format

8.java home配置

hadoop]# vi hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45

9.Start NameNode daemon and DataNode daemon:

hadoop-2.8.1]# sbin/start-dfs.sh 执行shell脚本

Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop000.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop000.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop000.out

/]# jps
9861 DataNode
9768 NameNode
10056 SecondaryNameNode
10492 Jps