当前位置: 代码迷 >> 综合 >> Sqoop 1.99.3 安装
  详细解决方案

Sqoop 1.99.3 安装

热度:57   发布时间:2023-12-10 21:11:23.0

  1. 1.安装准备工作:  
  2. 已经装好的hadoop环境是hadoop 2.2.0  
  3. 下载的sqoop安装包(注意是hadoop200)  
  4. http://www.us.apache.org/dist/sqoop/1.99.3/sqoop-1.99.3-bin-hadoop200.tar.gz  
  5. 2.解压文件到工作目录:  
  6. hadoop@hadoopMaster:$ sudo tar -xvf /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200.tar.gz   
  7. hadoop@hadoopMaster:mv /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200 /usr/local/sqoop  
  8. 3.修改环境变量:  
  9. hadoop@hadoopMaster:~$ vim /etc/profile  
  10. 添加如下内容:  
  11. #sqoop  
  12. export SQOOP_HOME=/usr/local/sqoop  
  13. export PATH=$SQOOP_HOME/bin:$PATH  
  14. export CATALINA_HOME=$SQOOP_HOME/server  
  15. export LOGDIR=$SQOOP_HOME/logs  
  16. 保存退出即时生效:  
  17. source /etc/profile  
  18. 4.修改sqoop配置:  
  19. hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/sqoop.properties   
  20. #修改指向我的hadoop安装目录  
  21. org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/  
  22. #把hadoop目录下的jar包都引进来  
  23. hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/catalina.properties   
  24. common.loader=/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar,/usr/local/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar  
  25. 5.下载mysql驱动包  
  26. mysql-connector-java-5.1.16-bin.jar  
  27. 6.启动/停止sqoop200  
  28. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh server start/stop  
  29. 查看启动日志:  
  30. hadoop@hadoopMaster:/usr/local/sqoop/server/logs$ vim catalina.out   
  31. 7.进入客户端交互目录  
  32. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client  
  33. +------------------------------------------+  
  34. |Sqoop home directory: /usr/local/sqoop    |  
  35. |Sqoop Shell: Type 'help' or '\h' for help.|  
  36. |sqoop:000>                                |  
  37. +------------------------------------------+  
  38. 为客户端配置服务器:  
  39. +---------------------------------------------------------------------+  
  40. |sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop|  
  41. |Server is set successfully                                           |  
  42. +---------------------------------------------------------------------+  
  43. 查版本信息:  
  44. +-----------------------------------------------------------------+  
  45. |sqoop:000> show version --all                                    |  
  46. |client version:                                                  |  
  47. |  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |  
  48. |  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |  
  49. |server version:                                                  |  
  50. |  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |  
  51. |  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |  
  52. |Protocol version:                                                |  
  53. |  [1]                                                            |  
  54. +-----------------------------------------------------------------+  
  55. 显示连接器:  
  56. +---------------------------------------------------------------------------------------------+  
  57. |sqoop:000> show connector --all                                                              |  
  58. |1 connector(s) to show:                                                                      |  
  59. |Connector with id 1:                                                                         |  
  60. |  Name: generic-jdbc-connector                                                               |  
  61. | Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector                                 |  
  62. |  Version: 1.99.3                                                                            |  
  63. |  Supported job types: [IMPORT, EXPORT]                                                      |  
  64. |    Connection form 1:                                                                       |  
  65. |      Name: connection                                                                       |  
  66. |      Label: Connection configuration                                                        |  
  67. |      Help: You must supply the information requested in order to create a connection object.|  
  68. |      Input 1:                                                                               |  
  69. |        .                                                                                    |  
  70. |        .                                                                                    |  
  71. |        .                                                                                    |  
  72. |   太长了,就拷贝这一点                                                                       |  
  73. +---------------------------------------------------------------------------------------------+  
  74. 创建数据库连接:  
  75. +---------------------------------------------------------------------------------------------+  
  76. |sqoop:000> create connection --cid 1                                                         |  
  77. |Creating connection for connector with id 1                                                  |  
  78. |Please fill following values to create new connection object                                 |  
  79. |Name: My first                                                                               |  
  80. |                                                                                             |  
  81. |Connection configuration                                                                     |  
  82. |                                                                                             |  
  83. |JDBC Driver Class: com.mysql.jdbc.Driver                                                     |  
  84. |JDBC Connection String: jdbc:mysql://localhost:3306/sqoop_stu                                |  
  85. |Username: root                                                                               |  
  86. |Password: **********                                                                         |  
  87. |JDBC Connection Properties:                                                                  |  
  88. |There are currently 0 values in the map:                                                     |  
  89. |entry#                                                                                       |  
  90. |                                                                                             |  
  91. |Security related configuration options                                                       |  
  92. |                                                                                             |  
  93. |Max connections: 100                                                                         |  
  94. |New connection was successfully created with validation status FINE and persistent id 1      |  
  95. +---------------------------------------------------------------------------------------------+  
  96. 创建导入任务  
  97. +------------------------------------------------------------------------------------+  
  98. |sqoop:001> create job --xid 1 --type import                                         |  
  99. |Creating job for connection with id 1                                               |  
  100. |Please fill following values to create new job object                               |  
  101. |Name: First job                                                                     |  
  102. |                                                                                    |  
  103. |Database configuration                                                              |  
  104. |                                                                                    |  
  105. |Schema name: traceweb                                                               |  
  106. |Table name: trace_web_application                                                   |  
  107. |Table SQL statement:                                                                |  
  108. |Table column names:                                                                 |  
  109. |Partition column name:                                                              |  
  110. |Nulls in partition column:                                                          |  
  111. |Boundary query:                                                                     |  
  112. |                                                                                    |  
  113. |Output configuration                                                                |  
  114. |                                                                                    |  
  115. |Storage type:                                                                       |  
  116. |  0 : HDFS                                                                          |  
  117. |Choose: 0                                                                           |  
  118. |Output format:                                                                      |  
  119. |  0 : TEXT_FILE                                                                     |  
  120. |  1 : SEQUENCE_FILE                                                                 |  
  121. |Choose: 1                                                                           |  
  122. |Compression format:                                                                 |  
  123. |  0 : NONE                                                                          |  
  124. |  1 : DEFAULT                                                                       |  
  125. |  2 : DEFLATE                                                                       |  
  126. |  3 : GZIP                                                                          |  
  127. |  4 : BZIP2                                                                         |  
  128. |  5 : LZO                                                                           |  
  129. |  6 : LZ4                                                                           |  
  130. |  7 : SNAPPY                                                                        |  
  131. |Choose: 0                                                                           |  
  132. |Output directory: /opt/sqoop_output                                                 |  
  133. |                                                                                    |  
  134. |Throttling resources                                                                |  
  135. |                                                                                    |  
  136. |Extractors:                                                                         |  
  137. |Loaders:                                                                            |  
  138. |New job was successfully created with validation status FINE  and persistent id 1   |  
  139. +------------------------------------------------------------------------------------+  
  140. 启动job:  
  141. +------------------------------------------------  
  142. |sqoop:000>  start job --jid 1    
  143. +------------------------------------------------  
  144.   
  145. 查看导入状态:  
  146. +------------------------------------------------  
  147. |sqoop:000>  status job --jid 1    
  148. |Submission details  
  149. |Job ID: 1  
  150. |Server URL: http://hadoopMaster:12000/sqoop/  
  151. |Created by: hadoop  
  152. |Creation date: 2014-05-23 18:51:05 CST  
  153. |Lastly updated by: hadoop  
  154. |External ID: job_local1566994033_0001  
  155. |   http://localhost:8080/  
  156. |2014-05-23 18:51:35 CST: UNKNOWN   
  157. +------------------------------------------------  
  158.   
  159. 查看输出目录:  
  160. +--------------------------------------------------------------------+  
  161. hadoop@hadoopMaster:~$ l /opt/sqoop_output/  
  162. 总用量 92  
  163. drwxrwxr-x 2 hadoop hadoop 4096  5月 23 18:52 .  
  164. drwxr-xr-x 8 hadoop hadoop 4096  5月 23 18:51 ..  
  165. -rw-r--r-- 1 hadoop hadoop  209  5月 23 18:51 part-m-00000.seq  
  166. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00000.seq.crc  
  167. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00001.seq  
  168. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00001.seq.crc  
  169. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00002.seq  
  170. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00002.seq.crc  
  171. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00003.seq  
  172. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00003.seq.crc  
  173. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00004.seq  
  174. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00004.seq.crc  
  175. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00005.seq  
  176. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00005.seq.crc  
  177. -rw-r--r-- 1 hadoop hadoop  207  5月 23 18:51 part-m-00006.seq  
  178. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00006.seq.crc  
  179. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00007.seq  
  180. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00007.seq.crc  
  181. -rw-r--r-- 1 hadoop hadoop  206  5月 23 18:51 part-m-00008.seq  
  182. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00008.seq.crc  
  183. -rw-r--r-- 1 hadoop hadoop  682  5月 23 18:51 part-m-00009.seq  
  184. -rw-rw-r-- 1 hadoop hadoop   16  5月 23 18:51 .part-m-00009.seq.crc  
  185. -rw-r--r-- 1 hadoop hadoop    0  5月 23 18:51 _SUCCESS  
  186. -rw-rw-r-- 1 hadoop hadoop    8  5月 23 18:51 ._SUCCESS.crc  
  187. +--------------------------------------------------------------------  
  188. sqoop:000> show job  
  189. +----+------------+--------+-----------+---------+  
  190. | Id |    Name    |  Type  | Connector | Enabled |  
  191. +----+------------+--------+-----------+---------+  
  192. 1  | First job  | IMPORT | 1         | true    |  
  193. 2  | importHDFS | IMPORT | 1         | true    |  
  194. +----+------------+--------+-----------+---------+  
  195. sqoop:000> delete job --jid 1  
  196. sqoop:000> show job            
  197. +----+------------+--------+-----------+---------+  
  198. | Id |    Name    |  Type  | Connector | Enabled |  
  199. +----+------------+--------+-----------+---------+  
  200. 2  | importHDFS | IMPORT | 1         | true    |  
  201. +----+------------+--------+-----------+---------+  
  202. sqoop:000> delete job --jid 2  
  203. sqoop:000> show job  
  204. +----+------+------+-----------+---------+  
  205. | Id | Name | Type | Connector | Enabled |  
  206. +----+------+------+-----------+---------+  
  207. +----+------+------+-----------+---------+  
  208.   
  209. sqoop:000> show connection  
  210.   
  211. 批处理模式:  
  212. sqoop.sh client /opt/sqoop/script.sqoop  
  213. hadoop@hadoopMaster:$ vim  /opt/sqoop/script.sqoop  
  214. #指定服务器信息  
  215. set server --host hadoopMaster --port 12000 --webapp sqoop  
  216. #执行JOB  
  217. start job --jid 1   
  218. +--------------------------------------------------------------------+  
  219. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client /opt/hadoop/mysql/batchModel.sqoop   
  220. Sqoop home directory: /usr/local/sqoop  
  221. sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop  
  222. Server is set successfully  
  223. sqoop:000> start job --jid 1  
  224. Submission details  
  225. Job ID: 1  
  226. Server URL: http://hadoopMaster:12000/sqoop/  
  227. Created by: hadoop  
  228. Creation date: 2014-05-30 10:55:10 CST  
  229. Lastly updated by: hadoop  
  230. External ID: job_local945860799_0003  
  231.     http://localhost:8080/  
  232. 2014-05-30 10:55:10 CST: BOOTING  - Progress is not available  
  233. +--------------------------------------------------------------------+  
  234.   
  235.   
  236.   
  237. https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Quickstart#Sqoop2Quickstart-Fullimportdemo  
  238.   
  239. ================================MYSQL=======================================  
  240. hadoop@hadoopMaster:~$ mysql -uroot -pjava  
  241. mysql> create database sqoop_stu;  
  242. Query OK, 1 row affected (0.03 sec)  
  243. mysql> use sqoop_stu;  
  244. Database changed  
  245. mysql> create table student(id int(3) auto_increment not null primary key, name char(10) not null, address varchar(50));  
  246. Query OK, 0 rows affected (0.41 sec)  
  247. mysql> insert into student values(1'Tom','beijing'),(2'Joan','shanghai'), (3'Wang''shenzheng');  
  248. Query OK, 3 rows affected (0.07 sec)  
  249. Records: 3  Duplicates: 0  Warnings: 0  
  250.   
  251.   
  252.   
  253.   
  254.   
  255. CREATE TABLE `demo_blog` (`id` int(11) NOT NULL AUTO_INCREMENT,  `blog` varchar(100) NOT NULL,  PRIMARY KEY (`id`)) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  
  256. CREATE TABLE `demo_log` (`operator` varchar(16) NOT NULL, `log` varchar(100) NOT NULL) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  
  257.   
  258.   
  259.   
  260.   
  261.   
  262.   
  263. https://hbase.apache.org/book/configuration.html#hadoop  
  264. http://www.tuicool.com/articles/NVfEVnn  

  相关解决方案