Ubuntu数据仓库环境部署(docker、tomcat、jdk、mysql、hadoop、hive、sqoop)

一、Docker安装（本文主要讲Ubuntu环境）

1.1 Centos Docker安装

# 镜像比较大, 需要准备一个网络稳定的环境
# 其中--mirror Aliyun代表使用阿里源
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

1.2 Ubuntu Docker安装【推荐】

安装curl程序

sudo apt install curl

下载一个安装脚本，再执行安装

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

1.3 MacOs Docker安装

# 下载安装包, 拖动安装即可
https://hub.docker.com/editions/community/docker-ce-desktop-mac/

1.4 Windows Docker安装【不推荐】

# win10家庭版 【参考】
https://docs.docker.com/docker-for-windows/install-windows-home/# win10专业版、商业版或教育版 【参考】
https://docs.docker.com/docker-for-windows/install/

二、容器准备

2.1 拉取镜像

给root配置密码

sudo passwd root

ubantu切换root用户

su root

拉取centos镜像

docker pull centos:7

查看docker信息

docker info

查看镜像

docker images

查看docker容器使用状态

docker ps -a

2.2 启动并创建容器

docker run -itd --privileged --name singleNode -h singleNode \
-p 2222:22 \
-p 3306:3306 \
-p 50070:50070 \
-p 8088:8088 \
-p 8080:8080 \
-p 10000:10000 \
-p 60010:60010 \
-p 9092:9092 \
centos:7 /usr/sbin/init

2.3 进入容器

docker exec -it singleNode /bin/bash

三、环境准备

3.1 安装必要软件

yum clean all
yum -y install unzip bzip2-devel vim bashname

3.2 配置SSH免密登录

#修改root密码
passwd root #输入两次密码
#安装必要ssh服务
yum install -y openssh openssh-server openssh-clients openssl openssl-devel 
#生成密钥
ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
方式二：ssh-copy-id singleNode
# 启动SSH服务
systemctl start sshd

3.3 设置时区

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3.4 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

3.5 安装包准备

java、MySQL、Hadoop、hive、sqoop等安装包
链接: https://pan.baidu.com/s/1Y0y-iMjrRjT6qhP7qClH_A
提取码: p6t1

退出容器

exit

在Ubuntu的root用户下创建文件夹/opt/software来存放各安装包

mkdir -p /opt/software

将root用户下的/opt/software中的各安装包拷贝到容器id为6d7e0d051c95的/opt文件夹下

docker cp /opt/software 6d7e0d051c95:/opt

进入容器

docker exec -it singleNode /bin/bash

四、MySQL安装

4.1 上传解压安装包

cd /opt/software/
tar xvf MySQL-5.5.40-1.linux2.6.x86_64.rpm-bundle.tar

4.2 安装必要依赖

yum -y install libaio perl

4.3 安装服务端和客户端

rpm -ivh MySQL-server-5.5.40-1.linux2.6.x86_64.rpm
rpm -ivh MySQL-client-5.5.40-1.linux2.6.x86_64.rpm 

4.4 启动并配置MySQL

systemctl start mysql
/usr/bin/mysqladmin -u root password 'root'
mysql -uroot -proot 
> update mysql.user set host='%' where host='localhost';
> delete from mysql.user where host<>'%' or user='';
> flush privileges;

五、安装JDK

5.1 上传并解压

tar zxvf /opt/software/jdk-8u171-linux-x64.tar.gz -C /opt/install/
ln -s /opt/install/jdk1.8.0_171 /opt/install/java

5.2 配置环境变量

vi /etc/profile
-------------------------------------------
export JAVA_HOME=/opt/install/java
export PATH=$JAVA_HOME/bin:$PATH
-------------------------------------------

5.3 查看版本

java -version

六、Hadoop安装

6.1 上传并解压

tar zxvf /opt/software/hadoop-2.6.0-cdh5.14.2.tar_2.gz -C /opt/install/
ln -s /opt/install/hadoop-2.6.0-cdh5.14.2 /opt/install/hadoop

6.2 修改配置

# 进入路径
cd /opt/install/hadoop/etc/hadoop/

6.2.1 配置core-site.xml

vi core-site.xml
-------------------------------------------
<configuration><property><name>fs.defaultFS</name><value>hdfs://singleNode:9000</value></property><property><name>hadoop.tmp.dir</name><value>/opt/install/hadoop/data/tmp</value></property>
</configuration>
-------------------------------------------

6.2.2 配置hdfs-site.xml

vi hdfs-site.xml
-------------------------------------------
<configuration><property><name>dfs.replication</name><value>1</value></property>
</configuration>
-------------------------------------------

6.2.3 配置mapred-site.xml

vi mapred-site.xml
-------------------------------------------
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapreduce.jobhistory.address</name><value>singleNode:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>singleNode:19888</value></property>
</configuration>
-------------------------------------------

6.2.4 配置yarn-site.xml

vi yarn-site.xml
-------------------------------------------
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.hostname</name><value>singleNode</value></property><property><name>yarn.log-aggregation-enable</name><value>true</value></property><property><name>yarn.log-aggregation.retain-seconds</name><value>604800</value></property>
</configuration>
-------------------------------------------

6.2.5 配置hadoop-env.sh

vi hadoop-env.sh
-------------------------------------------
export JAVA_HOME=/opt/install/java
-------------------------------------------

6.2.6 配置mapred-env.sh

vi mapred-env.sh
-------------------------------------------
export JAVA_HOME=/opt/install/java
-------------------------------------------

6.2.7 配置yarn-env.sh

vi yarn-env.sh
-------------------------------------------
export JAVA_HOME=/opt/install/java
-------------------------------------------

6.2.8 配置slaves

vi slaves
-------------------------------------------
singleNode
-------------------------------------------

6.3 添加环境变量

export HADOOP_HOME=/opt/install/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$HADOOP_HOME/bin:$PATH

6.4 HDFS格式化

hdfs namenode -format

6.5 启动Hadoop服务

$HADOOP_HOME/sbin/start-all.sh

6.6 Web端查看

查看50070端口

查看8088端口

七、Hive安装

7.1 上传并解压

tar zxvf /opt/software/hive-1.1.0-cdh5.14.2.tar.gz -C /opt/install/
ln -s /opt/install/hive-1.1.0-cdh5.14.2 /opt/install/hive

7.2 修改配置

# 进入路径
cd /opt/install/hive/conf/

7.2.1 修改hive-site.xml

vi hive-site.xml
-------------------------------------------
<configuration><property><name>hive.metastore.warehouse.dir</name><value>/home/hadoop/hive/warehouse</value></property><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://singleNode:3306/hive?createDatabaseIfNotExist=true</value></property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value></property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value></property><property><name>javax.jdo.option.ConnectionPassword</name><value>root</value></property><property><name>hive.exec.scratchdir</name><value>/home/hadoop/hive/data/hive-${user.name}</value><description>Scratch space for Hive jobs</description></property><property><name>hive.exec.local.scratchdir</name><value>/home/hadoop/hive/data/${user.name}</value><description>Local scratch space for Hive jobs</description></property>
</configuration>
-------------------------------------------

7.2.2 修改hive-env.sh

vi hive-env.sh
-------------------------------------------
HADOOP_HOME=/opt/install/hadoop
-------------------------------------------

7.3 添加依赖包

cp /opt/software/mysql-connector-java-5.1.31.jar /opt/install/hive/lib/

7.4 添加环境变量

export HIVE_HOME=/opt/install/hive
export PATH=$HIVE_HOME/bin:$PATH

7.5 启动服务

nohup hive --service metastore &
nohup hive --service hiveserver2 &

7.6 Jps查看

八、Sqoop安装

8.1 上传并解压

tar zxvf /opt/software/sqoop-1.4.6-cdh5.14.2.tar.gz -C /opt/install/
ln -s /opt/install/sqoop-1.4.6-cdh5.14.2 /opt/install/sqoop

8.2 修改sqoop-env.sh

cd /opt/install/sqoop/conf/
vi sqoop-env.sh
-------------------------------------------
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/install/hadoop#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/install/hadoop#Set the path to where bin/hive is available
export HIVE_HOME=/opt/install/hive
-------------------------------------------

8.3 添加依赖包

cp /opt/software/mysql-connector-java-5.1.31.jar /opt/install/sqoop/lib/
cp /opt/software/java-json.jar /opt/install/sqoop/lib/

8.4 添加环境变量

export SQOOP_HOME=/opt/install/sqoop
export PATH=$SQOOP_HOME/bin:$PATH

8.5 查看版本

sqoop version

九、安装Tomcat

9.1 下载Tomcat镜像

查找 Docker Hub 上的 tomcat 镜像

docker search tomcat

拉取官方最新版本的 tomcat 镜像：

docker pull tomcat

等待下载完成后，我们就可以在本地镜像列表里查到 REPOSITORY 为 tomcat 的镜像：

docker images|grep tomcat

9.2 使用Tomcat镜像

运行镜像，生成容器
使用 docker run 命令运行镜像：

docker run --name my-tomcat -p 8081:8080 -v $PWD/webapps/test:/usr/local/tomcat/webapps/test -d tomcat  

命令说明

-p 8081:8080：将主机的 8081 端口映射到容器的 8080 端口-v $PWD/webapps/test:/usr/local/tomcat/webapps/test：将主机中当前目录下的 /webapps/test 挂载到容器的/test

查看容器启动情况：

docker ps

查看容器中 tomcat 的启动日志：

docker logs -f my-tomcat

通过浏览器访问 tomcat：

http://个人机器IP:8081