通过hadoop + hive搭建离线式的分析系统之快速搭建一览

色魔王子7

[iyunv@master soft]# hadoop jar /usr/big/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar wordcount /input/2.txt /output/v1　　17/11/24 20:32:21 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
　　17/11/24 20:32:21 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
　　17/11/24 20:32:21 INFO input.FileInputFormat: Total input files to process : 1
　　17/11/24 20:32:21 INFO mapreduce.JobSubmitter: number of splits:1
　　17/11/24 20:32:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1430356259_0001
　　17/11/24 20:32:22 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
　　17/11/24 20:32:22 INFO mapreduce.Job: Running job: job_local1430356259_0001
　　17/11/24 20:32:22 INFO mapred.LocalJobRunner: OutputCommitter set in config null
　　17/11/24 20:32:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
　　17/11/24 20:32:22 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
　　17/11/24 20:32:22 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
　　17/11/24 20:32:22 INFO mapred.LocalJobRunner: Waiting for map tasks
　　17/11/24 20:32:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1430356259_0001_m_000000_0
　　17/11/24 20:32:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
　　17/11/24 20:32:22 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
　　17/11/24 20:32:22 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
　　17/11/24 20:32:22 INFO mapred.MapTask: Processing split: hdfs://192.168.23.196:9000/input/2.txt:0+40000002
　　17/11/24 20:32:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
　　17/11/24 20:32:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
　　17/11/24 20:32:22 INFO mapred.MapTask: soft limit at 83886080
　　17/11/24 20:32:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
　　17/11/24 20:32:22 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

　　17/11/24 20:32:22 INFO mapred.MapTask: Map output collector>　　17/11/24 20:32:23 INFO mapreduce.Job: Job job_local1430356259_0001 running in uber mode : false
　　17/11/24 20:32:23 INFO mapreduce.Job:  map 0% reduce 0%
　　17/11/24 20:32:23 INFO input.LineRecordReader: Found UTF-8 BOM and skipped it
　　17/11/24 20:32:27 INFO mapred.MapTask: Spilling map output
　　17/11/24 20:32:27 INFO mapred.MapTask: bufstart = 0; bufend = 27962024; bufvoid = 104857600
　　17/11/24 20:32:27 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 12233388(48933552); length = 13981009/6553600
　　17/11/24 20:32:27 INFO mapred.MapTask: (EQUATOR) 38447780 kvi 9611940(38447760)
　　17/11/24 20:32:32 INFO mapred.MapTask: Finished spill 0
　　17/11/24 20:32:32 INFO mapred.MapTask: (RESET) equator 38447780 kv 9611940(38447760) kvi 6990512(27962048)
　　17/11/24 20:32:33 INFO mapred.MapTask: Spilling map output
　　17/11/24 20:32:33 INFO mapred.MapTask: bufstart = 38447780; bufend = 66409804; bufvoid = 104857600
　　17/11/24 20:32:33 INFO mapred.MapTask: kvstart = 9611940(38447760); kvend = 21845332(87381328); length = 13981009/6553600
　　17/11/24 20:32:33 INFO mapred.MapTask: (EQUATOR) 76895558 kvi 19223884(76895536)
　　17/11/24 20:32:34 INFO mapred.LocalJobRunner: map > map
　　17/11/24 20:32:34 INFO mapreduce.Job:  map 67% reduce 0%
　　17/11/24 20:32:38 INFO mapred.MapTask: Finished spill 1
　　17/11/24 20:32:38 INFO mapred.MapTask: (RESET) equator 76895558 kv 19223884(76895536) kvi 16602456(66409824)
　　17/11/24 20:32:39 INFO mapred.LocalJobRunner: map > map
　　17/11/24 20:32:39 INFO mapred.MapTask: Starting flush of map output
　　17/11/24 20:32:39 INFO mapred.MapTask: Spilling map output
　　17/11/24 20:32:39 INFO mapred.MapTask: bufstart = 76895558; bufend = 100971510; bufvoid = 104857600
　　17/11/24 20:32:39 INFO mapred.MapTask: kvstart = 19223884(76895536); kvend = 7185912(28743648); length = 12037973/6553600
　　17/11/24 20:32:40 INFO mapred.LocalJobRunner: map > sort
　　17/11/24 20:32:43 INFO mapred.MapTask: Finished spill 2
　　17/11/24 20:32:43 INFO mapred.Merger: Merging 3 sorted segments

　　17/11/24 20:32:43 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total>　　17/11/24 20:32:43 INFO mapred.Task: Task:attempt_local1430356259_0001_m_000000_0 is done. And is in the process of committing
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: map > sort
　　17/11/24 20:32:43 INFO mapred.Task: Task 'attempt_local1430356259_0001_m_000000_0' done.
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local1430356259_0001_m_000000_0
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: map task executor complete.
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: Waiting for reduce tasks
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: Starting task: attempt_local1430356259_0001_r_000000_0
　　17/11/24 20:32:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
　　17/11/24 20:32:43 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
　　17/11/24 20:32:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
　　17/11/24 20:32:43 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@f8eab6f
　　17/11/24 20:32:43 INFO mapreduce.Job:  map 100% reduce 0%
　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=1336252800, maxSingleShuffleLimit=334063200, mergeThreshold=881926912, ioSortFactor=10, memToMemMergeOutputsThreshold=10
　　17/11/24 20:32:43 INFO reduce.EventFetcher: attempt_local1430356259_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
　　17/11/24 20:32:43 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1430356259_0001_m_000000_0 decomp: 60002 len: 60006 to MEMORY
　　17/11/24 20:32:43 INFO reduce.InMemoryMapOutput: Read 60002 bytes from map-output for attempt_local1430356259_0001_m_000000_0

　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of>　　17/11/24 20:32:43 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
　　17/11/24 20:32:43 INFO mapred.Merger: Merging 1 sorted segments

　　17/11/24 20:32:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total>　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: Merged 1 segments, 60002 bytes to disk to satisfy reduce memory limit
　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: Merging 1 files, 60006 bytes from disk
　　17/11/24 20:32:43 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
　　17/11/24 20:32:43 INFO mapred.Merger: Merging 1 sorted segments

　　17/11/24 20:32:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total>　　17/11/24 20:32:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
　　17/11/24 20:32:43 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
　　17/11/24 20:32:44 INFO mapred.Task: Task:attempt_local1430356259_0001_r_000000_0 is done. And is in the process of committing
　　17/11/24 20:32:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
　　17/11/24 20:32:44 INFO mapred.Task: Task attempt_local1430356259_0001_r_000000_0 is allowed to commit now
　　17/11/24 20:32:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1430356259_0001_r_000000_0' to hdfs://192.168.23.196:9000/output/v1/_temporary/0/task_local1430356259_0001_r_000000
　　17/11/24 20:32:44 INFO mapred.LocalJobRunner: reduce > reduce
　　17/11/24 20:32:44 INFO mapred.Task: Task 'attempt_local1430356259_0001_r_000000_0' done.
　　17/11/24 20:32:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local1430356259_0001_r_000000_0
　　17/11/24 20:32:44 INFO mapred.LocalJobRunner: reduce task executor complete.
　　17/11/24 20:32:44 INFO mapreduce.Job:  map 100% reduce 100%
　　17/11/24 20:32:44 INFO mapreduce.Job: Job job_local1430356259_0001 completed successfully
　　17/11/24 20:32:44 INFO mapreduce.Job: Counters: 35
　　File System Counters
　　FILE: Number of bytes read=1087044
　　FILE: Number of bytes written=2084932
　　FILE: Number of read operations=0
　　FILE: Number of large read operations=0
　　FILE: Number of write operations=0
　　HDFS: Number of bytes read=80000004
　　HDFS: Number of bytes written=54000
　　HDFS: Number of read operations=13
　　HDFS: Number of large read operations=0
　　HDFS: Number of write operations=4
　　Map-Reduce Framework
　　Map input records=1
　　Map output records=10000000
　　Map output bytes=80000000
　　Map output materialized bytes=60006
　　Input split bytes=103
　　Combine input records=10018000
　　Combine output records=24000
　　Reduce input groups=6000
　　Reduce shuffle bytes=60006
　　Reduce input records=6000
　　Reduce output records=6000
　　Spilled Records=30000
　　Shuffled Maps =1
　　Failed Shuffles=0
　　Merged Map outputs=1
　　GC time elapsed (ms)=1770
　　Total committed heap usage (bytes)=1776287744
　　Shuffle Errors
　　BAD_ID=0
　　CONNECTION=0
　　IO_ERROR=0
　　WRONG_LENGTH=0
　　WRONG_MAP=0
　　WRONG_REDUCE=0
　　File Input Format Counters
　　Bytes Read=40000002
　　File Output Format Counters
　　Bytes Written=54000