当前位置: 代码迷 >> 综合 >> Hadoop伪分布式部署 HDFS (hadoop用户)
  详细解决方案

Hadoop伪分布式部署 HDFS (hadoop用户)

热度:43   发布时间:2023-11-13 10:13:40.0

环境要求java、ssh

  • 添加hadoop用户

    1. [root@hadoop- 01 ~] # useradd hadoop
    2. [root@hadoop- 01 ~] # vi /etc/sudoers
    3. # 找到root ALL=(ALL) ALL,添加
    4. hadoop ALL=(ALL) NOPASSWD:ALL
  • 上传并解压

    1. [root@hadoop-01 software]# rz #上传 hadoop-2 .8 .1 .tar .gz
    2. [root@hadoop-01 software]# tar -xzvf hadoop-2 .8 .1 .tar .gz
  • 设置环境变量

    1. [root@hadoop-01 software] # vi /etc/profile
    2. export HADOOP_HOME=/opt/software/hadoop
    3. export PATH= $HADOOP_HOME/bin: $HADOOP_HOME/sbin: $PROTOC_HOME/bin: $FINDBUGS_HOME/bin: $MAVEN_HOME/bin: $JAVA_HOME/bin: $PATH
    4. [root@hadoop-01 software] # source /etc/profile
  • 设置用户用户组

    1. [root@hadoop-01 software]# chown -R hadoop :hadoop hadoop-2 .8 .1 /*
    2. [root@hadoop-01 software]# chown -R hadoop:hadoop hadoop-2.8.1
  • 切换用户hadoop

    1. [root@hadoop-01 software]# su - hadoop
    2. [root@hadoop-01 hadoop]# ll
    3. total 32
    4. drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14 :24 bin
    5. drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14 :24 etc
    6. drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14 :24 include
    7. drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14 :24 lib
    8. drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13 :59 libexec
    9. drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13 :59 logs
    10. drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14 :24 sbin
    11. drwxrwxr-x. 4 hadoop hadoop 4096 Jun 2 14 :24 share
    12. # bin: 可执行文件
    13. # etc: 配置文件
    14. # sbin: shell脚本,启动关闭 hdfs, yarn
  • 配置文件

    1. [hadoop@hadoop-01 ~]# cd /opt/software/hadoop
    2. [hadoop@hadoop-01 hadoop]# vi etc/hadoop/core-site.xml
    3. <configuration>
    4. <property>
    5. <name>fs.defaultFS </name>
    6. <value>hdfs://192.168.137.130:9000 </value> # 配置自己机器的IP
    7. </property>
    8. </configuration>
    9. [hadoop@hadoop-01 hadoop]# vi etc/hadoop/hdfs-site.xml
    10. <configuration>
    11. <property>
    12. <name>dfs.replication </name>
    13. <value>1 </value>
    14. </property>
    15. </configuration>
  • 配置hadoop用户的ssh信任关系

    1. # 公钥/密钥 配置无密码登录
    2. [hadoop@hadoop -01 ~] # ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    3. [hadoop@hadoop -01 ~] # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    4. [hadoop@hadoop -01 ~] # chmod 0600 ~/.ssh/authorized_keys
    5. # 查看日期,看是否配置成功
    6. [hadoop@hadoop -01 ~] # ssh hadoop-01 date
    7. The authenticity of host 'hadoop-01 (192.168.137.130)' can't be established.
    8. RSA key fingerprint is 09:f6: 4a:f1:a0:bd: 79:fd: 34:e7: 75: 94: 0b: 3c: 83: 5a.
    9. Are you sure you want to continue connecting (yes/no)? yes # 第一次回车输入yes
    10. Warning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts.
    11. Sun Aug 20 14: 22: 28 CST 2017
    12. [hadoop@hadoop -01 ~] # ssh hadoop-01 date #不需要回车输入yes,即OK
    13. Sun Aug 20 14: 22: 29 CST 2017
    14. [hadoop@hadoop -01 ~] # ssh localhost date
    15. The authenticity of host 'hadoop-01 (192.168.137.130)' can't be established.
    16. RSA key fingerprint is 09:f6: 4a:f1:a0:bd: 79:fd: 34:e7: 75: 94: 0b: 3c: 83: 5a.
    17. Are you sure you want to continue connecting (yes/no)? yes # 第一次回车输入yes
    18. Warning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts.
    19. Sun Aug 20 14: 22: 28 CST 2017
    20. [hadoop@hadoop -01 ~] # ssh localhost date #不需要回车输入yes,即OK
    21. Sun Aug 20 14: 22: 29 CST 2017
  • 格式化和启动

    1. [hadoop@hadoop -01 hadoop] # bin/hdfs namenode -format
    2. [hadoop@hadoop -01 hadoop] # sbin/start-dfs.sh
    3. ERROR:
    4. hadoop -01: Error: JAVA_HOME is not set and could not be found.
    5. localhost: Error: JAVA_HOME is not set and could not be found.
    6. 解决方法:添加环境变量
    7. [hadoop@hadoop -01 hadoop] # vi etc/hadoop/hadoop-env.sh
    8. # 将export JAVA_HOME=${JAVA_HOME}改为
    9. export JAVA_HOME=/usr/java/jdk1 .8.0_45
    10. [hadoop@hadoop -01 hadoop] # sbin/start-dfs.sh
    11. ERROR:
    12. mkdir: cannot create directory `/opt/software/hadoop -2.8.1/logs ': Permission denied
    13. 解决方法:
    14. [hadoop@hadoop -01 hadoop] # exit
    15. [root@hadoop -01 hadoop] # cd ../
    16. [root@hadoop -01 software] # chown -R hadoop:hadoop hadoop-2.8.1
    17. [root@hadoop -01 software] # su - hadoop
    18. [root@hadoop -01 ~] # cd /opt/software/hadoop
    19. # 继续启动
    20. [hadoop@hadoop -01 hadoop] # sbin/start-dfs.sh
  • 检查是否成功

    1. [hadoop@hadoop- 01 hadoop] # jps
    2. 19536 DataNode
    3. 19440 NameNode
    4. 19876 Jps
    5. 19740 SecondaryNameNode

  相关解决方案