当前位置: 代码迷 >> 综合 >> Sqoop 相关总结
  详细解决方案

Sqoop 相关总结

热度:36   发布时间:2023-11-18 05:19:29.0

1.sqoop命令的执行方式:

(1). Python : retCode = subprocess.call(sqoopCmd, shell=True)
eg:

    sqoopCmd = "sqoop import " \+ "--connect '" + dbConnStr + "' " \+ "--username '" + dbUser + "' " \+ "--password '" + dbPass + "' " \+ "--query \"" + sqoopQueryStr + "\" " \+ "--null-string '\\\\N' " \+ "--null-non-string '\\\\N' " \+ "--target-dir '" + sqoopTargetFile + "' " \+ "--hive-drop-import-delims " \+ "--fields-terminated-by '" + sqoopFieldsSep + "' " \+ "--lines-terminated-by '" + sqoopLinesSep + "' " \+ "--append " \+ "--temporary-rootdir '" + sqoopTmpDir + "' " \+ "--m " + sqoopMaps \logging.info("sqoop cmd is : %s" % (sqoopCmd))retCode = subprocess.call(sqoopCmd, shell=True)

(2). Shell : 编写shell脚本,直接将参数写入shell中

eg:

#!/bin/bash#CURR_DATE=`date +"%Y-%m-%d %H:%M:%S"`------>不能使用
v_sql="insert into origin_ennenergy_energytrade.test2 values('"$(date +"%Y-%m-%d %H:%M:%S")"','"Y"')"
echo $v_sql#insert into origin_ennenergy_energytrade.test2 values('2016-08-09 10:39:44','Y')hive -e "$v_sql;"
sqoop export --connect jdbc:mysql://ip:3306/test23?characterEncoding=utf8 --username root --password 123--table test2--export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/test2/* --input-fields-terminated-by "\t" --update-mode allowinsert --update-key times;

执行shell

sh test.sh

2.Sqoop参数详见:

http://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html

  相关解决方案