1.先去hadoop官网下载hadoop的源码?http://svn.apache.org/repos/asf/hadoop/common/trunk
2.下载maven3，当前hadoop的最新版必须使用maven3编译
3.到hadoop下载源码目录执行mvn clean install；mvn eclipse:eclipse；
4.将源码导入eclipse；
5.在eclipse设置执行的WordCount.java的jvm启动参数，最少需要两个，输入目录和输出目录
??
6.然后就可以设置断点进行调试了，我们在处理mapreduce的主干流程上设置断点
??org.apache.hadoop.mapred.LocalJobRunner这个类的run方法上
? 我们看到在我们设置的输入输出目录，然后使用默认的hadoop单机配置下，mapTask有16个，reduceTask有1个
我们先看看我们的输入目录，刚好是16个文件，说明每个输入文件默认启动一个mapTask
?
而reduce怎么是一个，怎么处理16个mapTask的输出呢
在org.apache.hadoop.mapred.ReduceTask这个reduce处理中run方法中会对所有的map输出做一个merge，然后作为reduceTask的输入
if (!isLocal) { Class combinerClass = conf.getCombinerClass(); CombineOutputCollector combineCollector = (null != combinerClass) ? new CombineOutputCollector(reduceCombineOutputCounter, reporter, conf) : null; Shuffle shuffle = new Shuffle(getTaskID(), job, FileSystem.getLocal(job), umbilical, super.lDirAlloc, reporter, codec, combinerClass, combineCollector, spilledRecordsCounter, reduceCombineInputCounter, shuffledMapsCounter, reduceShuffleBytes, failedShuffleCounter, mergedMapOutputsCounter, taskStatus, copyPhase, sortPhase, this, mapOutputFile); rIter = shuffle.run(); } else { // local job runner doesn't have a copy phase copyPhase.complete(); final FileSystem rfs = FileSystem.getLocal(job).getRaw(); rIter = Merger.merge(job, rfs, job.getMapOutputKeyClass(), job.getMapOutputValueClass(), codec, getMapFiles(rfs, true), !conf.getKeepFailedTaskFiles(), job.getInt(JobContext.IO_SORT_FACTOR, 100), new Path(getTaskID().toString()), job.getOutputKeyComparator(), reporter, spilledRecordsCounter, null, null); }?由于用到的是单机模式，所以没有用到Shuffle的过程。
?
整个日志输出如下：
2012-11-05 12:46:37,006 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - session.id is deprecated. Instead, use dfs.metrics.session-id2012-11-05 12:46:37,050 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=2012-11-05 12:46:37,195 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2012-11-05 12:46:37,263 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(247)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).2012-11-05 12:46:37,315 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 162012-11-05 12:46:37,784 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(368)) - number of splits:162012-11-05 12:46:37,832 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir2012-11-05 12:46:37,833 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class2012-11-05 12:46:37,834 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class2012-11-05 12:46:37,835 WARN [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir2012-11-05 12:46:38,048 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(438)) - Submitting tokens for job: job_local_00012012-11-05 12:46:38,597 INFO [main] mapreduce.Job (Job.java:submit(1222)) - The url to track the job: http://localhost:8080/2012-11-05 12:46:38,663 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1267)) - Running job: job_local_00012012-11-05 12:46:38,666 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(320)) - OutputCommitter set in config null2012-11-05 12:46:38,677 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(338)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter2012-11-05 12:46:39,666 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1288)) - Job job_local_0001 running in uber mode : false2012-11-05 12:46:39,668 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 0% reduce 0%2012-11-05 12:52:04,731 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000000_02012-11-05 12:52:04,894 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@16c46422012-11-05 12:52:04,903 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/capacity-scheduler.xml:0+74572012-11-05 12:52:05,006 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:05,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:05,062 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 10166; bufvoid = 1048576002012-11-05 12:52:05,063 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26211052(104844208); length = 3345/65536002012-11-05 12:52:05,151 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:05,155 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000000_0 is done. And is in the process of committing2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000000_0' done.2012-11-05 12:52:05,166 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000000_02012-11-05 12:52:05,852 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 0%2012-11-05 12:52:06,389 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000001_02012-11-05 12:52:06,389 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:run(386)) - Waiting for map tasks2012-11-05 12:52:06,394 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1a3c2bf2012-11-05 12:52:06,398 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-policy.xml:0+46442012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:06,466 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:06,467 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:06,467 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:06,483 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:06,483 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 6454; bufvoid = 1048576002012-11-05 12:52:06,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26212228(104848912); length = 2169/65536002012-11-05 12:52:06,508 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:06,515 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000001_0 is done. And is in the process of committing2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000001_0' done.2012-11-05 12:52:06,520 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000001_02012-11-05 12:52:06,521 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000002_02012-11-05 12:52:06,528 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@adc92c2012-11-05 12:52:06,529 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/log4j.properties:0+44412012-11-05 12:52:06,615 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:06,616 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:06,724 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 5492; bufvoid = 1048576002012-11-05 12:52:06,725 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213316(104853264); length = 1081/65536002012-11-05 12:52:06,737 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:06,741 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000002_0 is done. And is in the process of committing2012-11-05 12:52:06,743 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000002_0' done.2012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000002_02012-11-05 12:52:06,744 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000003_02012-11-05 12:52:06,747 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1c6f86d2012-11-05 12:52:06,749 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-env.sh:0+22372012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,288 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,289 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,289 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 3255; bufvoid = 1048576002012-11-05 12:52:07,322 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213356(104853424); length = 1041/65536002012-11-05 12:52:07,330 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,333 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000003_0 is done. And is in the process of committing2012-11-05 12:52:07,334 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000003_0' done.2012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000003_02012-11-05 12:52:07,335 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000004_02012-11-05 12:52:07,339 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@19d34ca2012-11-05 12:52:07,401 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/mapred-queue-acls.xml:0+20332012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,484 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,485 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 3020; bufvoid = 1048576002012-11-05 12:52:07,485 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26213292(104853168); length = 1105/65536002012-11-05 12:52:07,494 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,496 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000004_0 is done. And is in the process of committing2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000004_0' done.2012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000004_02012-11-05 12:52:07,498 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000005_02012-11-05 12:52:07,500 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1fe35152012-11-05 12:52:07,502 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hadoop-metrics2.properties:0+14882012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,560 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,566 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1710; bufvoid = 1048576002012-11-05 12:52:07,567 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214160(104856640); length = 237/65536002012-11-05 12:52:07,574 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,577 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000005_0 is done. And is in the process of committing2012-11-05 12:52:07,582 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,582 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000005_0' done.2012-11-05 12:52:07,583 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000005_02012-11-05 12:52:07,583 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000006_02012-11-05 12:52:07,585 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@4bc5a52012-11-05 12:52:07,587 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/ssl-client.xml.example:0+12432012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,697 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,698 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1530; bufvoid = 1048576002012-11-05 12:52:07,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214044(104856176); length = 353/65536002012-11-05 12:52:07,709 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,712 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000006_0 is done. And is in the process of committing2012-11-05 12:52:07,713 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000006_0' done.2012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000006_02012-11-05 12:52:07,714 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000007_02012-11-05 12:52:07,716 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@15f7c4b2012-11-05 12:52:07,718 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/ssl-server.xml.example:0+11952012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,783 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,788 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 1470; bufvoid = 1048576002012-11-05 12:52:07,789 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214060(104856240); length = 337/65536002012-11-05 12:52:07,793 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,795 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000007_0 is done. And is in the process of committing2012-11-05 12:52:07,796 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000007_0' done.2012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000007_02012-11-05 12:52:07,797 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000008_02012-11-05 12:52:07,798 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@17eb1942012-11-05 12:52:07,799 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/configuration.xsl:0+5352012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 666; bufvoid = 1048576002012-11-05 12:52:07,861 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/65536002012-11-05 12:52:07,866 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:07,869 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000008_0 is done. And is in the process of committing2012-11-05 12:52:07,870 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000008_0' done.2012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000008_02012-11-05 12:52:07,871 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000009_02012-11-05 12:52:07,873 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@14924fb2012-11-05 12:52:07,874 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/taskcontroller.cfg:0+3822012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:07,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:07,994 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 546; bufvoid = 1048576002012-11-05 12:52:08,001 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214236(104856944); length = 161/65536002012-11-05 12:52:08,007 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,010 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000009_0 is done. And is in the process of committing2012-11-05 12:52:08,011 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,011 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000009_0' done.2012-11-05 12:52:08,012 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000009_02012-11-05 12:52:08,012 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000010_02012-11-05 12:52:08,014 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1e541672012-11-05 12:52:08,016 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/fair-scheduler.xml:0+3272012-11-05 12:52:08,075 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,076 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 460; bufvoid = 1048576002012-11-05 12:52:08,085 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214256(104857024); length = 141/65536002012-11-05 12:52:08,088 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,091 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000010_0 is done. And is in the process of committing2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000010_0' done.2012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000010_02012-11-05 12:52:08,092 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000011_02012-11-05 12:52:08,094 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@ccde812012-11-05 12:52:08,095 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/core-site.xml:0+1782012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,154 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 1048576002012-11-05 12:52:08,157 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/65536002012-11-05 12:52:08,162 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,168 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000011_0 is done. And is in the process of committing2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000011_0' done.2012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000011_02012-11-05 12:52:08,170 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000012_02012-11-05 12:52:08,172 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@fbee672012-11-05 12:52:08,173 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/mapred-site.xml:0+1782012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,302 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,391 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 1048576002012-11-05 12:52:08,392 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/65536002012-11-05 12:52:08,398 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,402 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000012_0 is done. And is in the process of committing2012-11-05 12:52:08,404 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000012_0' done.2012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000012_02012-11-05 12:52:08,405 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000013_02012-11-05 12:52:08,407 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@ac82112012-11-05 12:52:08,409 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/hdfs-site.xml:0+1782012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,468 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,487 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 239; bufvoid = 1048576002012-11-05 12:52:08,488 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/65536002012-11-05 12:52:08,491 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,493 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000013_0 is done. And is in the process of committing2012-11-05 12:52:08,495 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000013_0' done.2012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000013_02012-11-05 12:52:08,496 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000014_02012-11-05 12:52:08,498 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@5a4a142012-11-05 12:52:08,500 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/masters:0+102012-11-05 12:52:08,558 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,559 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,562 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 14; bufvoid = 1048576002012-11-05 12:52:08,563 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/65536002012-11-05 12:52:08,565 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,568 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000014_0 is done. And is in the process of committing2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000014_0' done.2012-11-05 12:52:08,570 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000014_02012-11-05 12:52:08,571 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: attempt_local_0001_m_000015_02012-11-05 12:52:08,572 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@527e312012-11-05 12:52:08,574 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(699)) - Processing split: file:/home/weijianzhongwj/software/hadoop-1.1.0/conf/slaves:0+102012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1133)) - (EQUATOR) 0 kvi 26214396(104857584)2012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(929)) - mapreduce.task.io.sort.mb: 1002012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(930)) - soft limit at 838860802012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(931)) - bufstart = 0; bufvoid = 1048576002012-11-05 12:52:08,682 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:<init>(932)) - kvstart = 26214396; length = 65536002012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1395)) - Starting flush of map output2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1414)) - Spilling map output2012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1415)) - bufstart = 0; bufend = 14; bufvoid = 1048576002012-11-05 12:52:08,688 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1417)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/65536002012-11-05 12:52:08,693 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1603)) - Finished spill 02012-11-05 12:52:08,695 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_m_000015_0 is done. And is in the process of committing2012-11-05 12:52:08,698 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map2012-11-05 12:52:08,699 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_m_000015_0' done.2012-11-05 12:52:08,699 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(238)) - Finishing task: attempt_local_0001_m_000015_02012-11-05 12:52:08,699 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:run(394)) - Map task executor complete.2012-11-05 12:52:58,415 INFO [Thread-10] mapred.Task (Task.java:initialize(566)) - Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@e851022012-11-05 12:58:16,363 INFO [Thread-10] mapred.Merger (Merger.java:merge(549)) - Merging 16 sorted segments2012-11-05 12:59:41,406 INFO [Thread-10] mapred.Merger (Merger.java:merge(653)) - Merging 7 intermediate segments out of a total of 162012-11-05 12:59:44,605 INFO [Thread-10] mapred.Merger (Merger.java:merge(648)) - Down to the last merge-pass, with 10 segments left of total size: 22378 bytes2012-11-05 12:59:44,966 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - 2012-11-05 12:59:46,437 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce2012-11-05 12:59:46,486 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 66%2012-11-05 12:59:47,163 WARN [Thread-10] conf.Configuration (Configuration.java:warnOnceIfDeprecated(823)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords2012-11-05 12:59:47,198 INFO [Thread-10] mapred.Task (Task.java:done(980)) - Task:attempt_local_0001_r_000000_0 is done. And is in the process of committing2012-11-05 12:59:47,198 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce2012-11-05 12:59:47,198 INFO [Thread-10] mapred.Task (Task.java:commit(1141)) - Task attempt_local_0001_r_000000_0 is allowed to commit now2012-11-05 12:59:47,199 INFO [Thread-10] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(432)) - Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/weijianzhongwj/software/hadoop-1.1.0/out/_temporary/0/task_local_0001_r_0000002012-11-05 12:59:47,199 INFO [Thread-10] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce2012-11-05 12:59:47,200 INFO [Thread-10] mapred.Task (Task.java:sendDone(1100)) - Task 'attempt_local_0001_r_000000_0' done.2012-11-05 12:59:47,486 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) - map 100% reduce 100%2012-11-05 12:59:47,487 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1306)) - Job job_local_0001 completed successfully2012-11-05 12:59:47,602 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1313)) - Counters: 27 File System Counters FILE: Number of bytes read=667992 FILE: Number of bytes written=2711113 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=749 Map output records=2585 Map output bytes=35514 Map output materialized bytes=22535 Input split bytes=2154 Combine input records=2585 Combine output records=1089 Reduce input groups=789 Reduce shuffle bytes=0 Reduce input records=1089 Reduce output records=789 Spilled Records=2293 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=603 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=4293918720 File Input Format Counters Bytes Read=26536 File Output Format Counters Bytes Written=15413?
有很多的统计信息
?
我们再看下最终的输出目录：
?