错误日志
ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already exists
[root@mihaoyu151 ~]# sqoop import \
> --connect jdbc:mysql://mihaoyu151:3306/mybatisdb \
> --username root \
> --password root \
> --table student \
> --hive-table kb15.student_mysql \
> --hive-import \
> --m 1
Warning: /opt/soft/sqoop146/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/soft/sqoop146/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/11/23 09:19:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
21/11/23 09:19:56 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/11/23 09:19:56 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
21/11/23 09:19:56 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
21/11/23 09:19:56 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/11/23 09:19:56 INFO tool.CodeGenTool: Beginning code generation
21/11/23 09:19:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
21/11/23 09:19:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
21/11/23 09:19:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/soft/hadoop260
Note: /tmp/sqoop-root/compile/03e11c03cfa616424a2d39aed988c741/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
21/11/23 09:19:58 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/03e11c03cfa616424a2d39aed988c741/student.jar
21/11/23 09:19:58 WARN manager.MySQLManager: It looks like you are importing from mysql.
21/11/23 09:19:58 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
21/11/23 09:19:58 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
21/11/23 09:19:58 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
21/11/23 09:19:58 INFO mapreduce.ImportJobBase: Beginning import of student
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/11/23 09:19:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/11/23 09:19:58 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/11/23 09:19:59 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/11/23 09:19:59 INFO client.RMProxy: Connecting to ResourceManager at mihaoyu151/192.168.133.151:8032
21/11/23 09:19:59 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already exists
21/11/23 09:19:59 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already existsat org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196)at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169)at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266)at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)at org.apache.sqoop.Sqoop.run(Sqoop.java:143)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
解决办法
[root@mihaoyu151 ~]# hdfs dfs -rmr /user/root/student
rmr: DEPRECATED: Please use 'rm -r' instead.
21/11/23 09:21:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Deleted /user/root/student[root@mihaoyu151 ~]# sqoop import \
> --connect jdbc:mysql://mihaoyu151:3306/mybatisdb \
> --username root \
> --password root \
> --table student \
> --hive-table kb15.student_mysql \
> --hive-import \
> --m 1
Warning: /opt/soft/sqoop146/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/soft/sqoop146/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/11/23 09:21:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
21/11/23 09:21:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/11/23 09:21:24 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
21/11/23 09:21:24 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
21/11/23 09:21:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/11/23 09:21:24 INFO tool.CodeGenTool: Beginning code generation
21/11/23 09:21:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
21/11/23 09:21:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
21/11/23 09:21:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/soft/hadoop260
Note: /tmp/sqoop-root/compile/c2780f73626821864f7ba23e7758fe9d/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
21/11/23 09:21:26 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/c2780f73626821864f7ba23e7758fe9d/student.jar
21/11/23 09:21:26 WARN manager.MySQLManager: It looks like you are importing from mysql.
21/11/23 09:21:26 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
21/11/23 09:21:26 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
21/11/23 09:21:26 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
21/11/23 09:21:26 INFO mapreduce.ImportJobBase: Beginning import of student
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/11/23 09:21:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/11/23 09:21:27 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
21/11/23 09:21:27 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
21/11/23 09:21:27 INFO client.RMProxy: Connecting to ResourceManager at mihaoyu151/192.168.133.151:8032
21/11/23 09:21:29 INFO db.DBInputFormat: Using read commited transaction isolation
21/11/23 09:21:29 INFO mapreduce.JobSubmitter: number of splits:1
21/11/23 09:21:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637624153438_0005
21/11/23 09:21:29 INFO impl.YarnClientImpl: Submitted application application_1637624153438_0005
21/11/23 09:21:29 INFO mapreduce.Job: The url to track the job: http://mihaoyu151:8088/proxy/application_1637624153438_0005/
21/11/23 09:21:29 INFO mapreduce.Job: Running job: job_1637624153438_0005
21/11/23 09:21:37 INFO mapreduce.Job: Job job_1637624153438_0005 running in uber mode : false
21/11/23 09:21:37 INFO mapreduce.Job: map 0% reduce 0%
21/11/23 09:21:43 INFO mapreduce.Job: map 100% reduce 0%
21/11/23 09:21:43 INFO mapreduce.Job: Job job_1637624153438_0005 completed successfully
21/11/23 09:21:43 INFO mapreduce.Job: Counters: 30File System CountersFILE: Number of bytes read=0FILE: Number of bytes written=161559FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=87HDFS: Number of bytes written=331HDFS: Number of read operations=4HDFS: Number of large read operations=0HDFS: Number of write operations=2Job Counters Launched map tasks=1Other local map tasks=1Total time spent by all maps in occupied slots (ms)=2891Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=2891Total vcore-milliseconds taken by all map tasks=2891Total megabyte-milliseconds taken by all map tasks=2960384Map-Reduce FrameworkMap input records=9Map output records=9Input split bytes=87Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=40CPU time spent (ms)=590Physical memory (bytes) snapshot=117817344Virtual memory (bytes) snapshot=2731085824Total committed heap usage (bytes)=60882944File Input Format Counters Bytes Read=0File Output Format Counters Bytes Written=331
21/11/23 09:21:43 INFO mapreduce.ImportJobBase: Transferred 331 bytes in 15.5214 seconds (21.3254 bytes/sec)
21/11/23 09:21:43 INFO mapreduce.ImportJobBase: Retrieved 9 records.
21/11/23 09:21:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
21/11/23 09:21:43 INFO hive.HiveImport: Loading uploaded data into Hive
21/11/23 09:21:45 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
21/11/23 09:21:45 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
21/11/23 09:21:45 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
21/11/23 09:21:45 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
21/11/23 09:21:45 INFO hive.HiveImport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/11/23 09:21:45 INFO hive.HiveImport: 2021-11-23 09:21:45,188 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/11/23 09:21:46 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
21/11/23 09:21:46 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
21/11/23 09:21:46 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
21/11/23 09:21:46 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
21/11/23 09:21:46 INFO hive.HiveImport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
21/11/23 09:21:47 INFO hive.HiveImport: 21/11/23 09:21:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/11/23 09:21:47 INFO hive.HiveImport: 21/11/23 09:21:47 WARN conf.HiveConf: HiveConf of name hive.server2.thrift.client.user does not exist
21/11/23 09:21:47 INFO hive.HiveImport: 21/11/23 09:21:47 WARN conf.HiveConf: HiveConf of name hive.server2.thrift.client.password does not exist
21/11/23 09:21:47 INFO hive.HiveImport:
21/11/23 09:21:47 INFO hive.HiveImport: Logging initialized using configuration in file:/opt/soft/hive110/conf/hive-log4j.properties
21/11/23 09:21:53 INFO hive.HiveImport: OK
21/11/23 09:21:53 INFO hive.HiveImport: Time taken: 4.928 seconds
21/11/23 09:21:53 INFO hive.HiveImport: Loading data to table kb15.student_mysql
21/11/23 09:21:54 INFO hive.HiveImport: Table kb15.student_mysql stats: [numFiles=1, totalSize=331]
21/11/23 09:21:54 INFO hive.HiveImport: OK
21/11/23 09:21:54 INFO hive.HiveImport: Time taken: 0.716 seconds
21/11/23 09:21:54 INFO hive.HiveImport: Hive import complete.
21/11/23 09:21:54 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.