当前位置: 代码迷 >> 综合 >> Hadoop伪分布式模式部署 HDFS(root用户)
  详细解决方案

Hadoop伪分布式模式部署 HDFS(root用户)

热度:73   发布时间:2023-11-13 10:14:32.0

Hadoop伪分布式模式部署

Hadoop2.x:

官网:hadoop.apache.org
三个组件:
HDFS:分布式文件系统,存储
MapReduce:分布式计算
Yarn:资源(cpu + memory)和JOB调度监控

文档地址:

    http://hadoop.apache.org/docs/r2.8.2/

部署方式:
1.单机模式(Standalone):1个Java进程
2.伪分布模式(Pseudo-Distribute Mode):开发|学习   多个Java进程
3.集群模式(Cluster Mode):生产 多态机器多个Java进程

伪分布式部署:HDFS
1.创建hadoop服务的一个用户
# useradd hadoop
# id hadoop
# vi /etc/sudoers
    hadoop ALL=(root) NOPASSWD:ALL


2.部署Java
使用:Oracle jdk1.8
尽力不要使用: Open JDk
    2.1、解压+环境变量
    2.2、CDH课 /usr/java
        最好:which java

3.部署ssh,确保其是运行的

查看:默认是已经安装
    service sshd status

4.解压Hadoop
解压:
    software]# tar -zxvf hadoop-2.8.1.tar.gz
修改权限:
    chown -R root:root hadoop-2.8.1

配置环境变量
    vi /etc/profile
    export HADOOP_HOME=/opt/software/hadoop-2.8.1
    export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
    # which hadoop
    # /opt/software/hadoop-2.8.1/bin/hadoop
软连接:
    software]# ln -s /opt/software/hadoop-2.8.1 hadoop
修改用户和用户组:
    software]# chown -R hadoop:hadoop hadoop
        修改软连接
    software]# chown -R hadoop:hadoop hadoop/*
        修改软连接文件夹里面的内容
    software]# chown -R hadoop:hadoop hadoop-2.8.1
        修改原文件夹
 
    chown -R hadoop:hadoop 文件夹:修改文件夹和文件夹里面的内容
    chown -R hadoop:hadoop 软连接文件夹:修改软连接文件夹,不会修改软连接文件夹里面的
    chown -R hadoop:hadoop/* 软连接文件夹/*:软连接文件夹不修改,只修改软连接文件夹里面的
   
    software]# cd hadoop-2.8.1
    #删除文件夹内.cmd后缀文件避免混淆
    hadoop-2.8.1]# rm -rf /hadoop-2.8.1/bin/*.cmd
    hadoop-2.8.1]# rm -rf /hadoop-2.8.1/sbin/*.cmd

        bin:执行命令的shell
        etc:配置文件
        lib:lib库
        sbin:启动和关闭hadoop里面的组件(进程)
        hadoop-2.8.1/share/hadoop/hdfs:jar包

5.hadoop的模块组件配置core-site.xml
Hadoop-env.sh :hadoop配合环境
core-site.xml     :hadoop 核心配置文件
hdfs-site.xml     :针对于hdfs服务的会起进程
[mapred-site.xml:MapReduce计算所需要的配置文件] 只有在jar计算时才有
yarn-site.xml     :yarn服务的会起进程
slaves                :集群的机器名称

配置core-site文件

etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

6.配置root用户的ssh的信任关系

~]# rm -rf .ssh
~]# ssh-keygen
~]# cd .ssh
 .ssh]# cat id_rsa.pub >> authorized_keys 添加自己的公钥到信任名单
#第一次执行
 ~]# ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Sun May 13 21:49:14 CST 2018
#第二次执行
~]# ssh localhost date
Sun May 13 21:49:17 CST 2018

7.格式化文件系统

   $ bin/hdfs namenode -format

8.java home配置

hadoop]# vi  hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_45

9.Start NameNode daemon and DataNode daemon:

hadoop-2.8.1]# sbin/start-dfs.sh  执行shell脚本

Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop000.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop000.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop000.out

 /]# jps
9861 DataNode
9768 NameNode
10056 SecondaryNameNode
10492 Jps


  相关解决方案