1、技術創新，變革未來Hadoop與數據分析概述OutlineHadoop基本概念Hadoop的應用范圍Hadoop底層實現原理Hive與數據分析Hadoop集群管理典型的Hadoop離線分析系統架構常見問題及解決方案關于打撲克的哲學打撲克與MapReduceInput split shuffle output 分牌各自齊牌交換再次理牌搞定統計單詞數The weather is goodThis guyis a good manToday is goodGood manis goodthe 1weather 1is 1good 1today 1is 1good 1this 1guy 1is 1a

2、1good 1man 1good 1man 1is 1good 1a 1 good 1good 1good 1good 1good 1man 1man 1the 1weather 1today 1guy 1is 1is 1is 1is 1this 1a 1good 5guy 1is 4man 2the 1this 1today 1weather 1流量計算6趨勢分析7/截圖用戶推薦8分布式索引910Hadoop 核心Hadoop Common分布式文件系統HDFSMapReduce框架并行數據分析語言Pig 列存儲NoSQL數據庫 Hbase分布式協調器Zookeeper數據倉庫Hive(使用

3、SQL)Hadoop日志分析工具ChukwaHadoop生態系統11DataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data da

4、taData data data data dataResultsData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataHadoop ClusterDFS Block 1DFS Block 1DFS Block 2DFS Block 2DFS Block 2DFS Block 1DFS Block 3DFS B

5、lock 3DFS Block 3MAPMAPMAPReduceHadoop實現作業執行流程/ MapClass1中的map方法 public void map(LongWritable Key, Text value, OutputCollector output, Reporter reporter) throws IOException String strLine = value.toString(); String strList = strLine.split(); String mid = strList3; String sid = strList4;String timest

6、r = strList0;try timestr = timestr.substring(0,10);catch(Exception e)return;timestr += 0000; / 省略數十行 output.collect(new Text(mid + “” + “sid” + timestr , .);Hadoop案例(1) public static class Reducer1 extends MapReduceBase implements Reducer private Text word = new Text(); private Text str = new Text()

7、; public void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException String t = key.toString().split(); word.set(t0);/ str.set(t1); output.collect(word,str);/uid kind /reduce /Reduce0b Hadoop案例(2) public static class MapClass2 extends MapReduceBase implements

8、 Mapper private Text word = new Text(); private Text str = new Text(); public void map(LongWritable Key, Text value, OutputCollector output, Reporter reporter) throws IOException String strLine = value.toString(); String strList = strLine.split(s+);word.set(strList0);str.set(strList1);output.collect

9、(word,str); Hadoop案例(3) public static class Reducer2 extends MapReduceBase implements Reducer private Text word = new Text(); private Text str = new Text(); public void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException while(values.hasNext() String t = v

10、alues.next().toString(); / 省略數十行代碼 / 省略數十行代碼 output.collect(new Text(mid + “” + sid + “”) + ., .) Hadoop案例(4)BADAACBCBCDGroupCo-groupFunctionAggregate FilterFilterThinking in MapReduce(1)Thinking in MapReduce(2)Magics of Hive:SELECT COUNT(DISTINCT mid) FROM log_tableHive的魔力為什么淘寶采用Hadoop?webalizeraws

11、tat般若Atpanel時代日志最高達250GB/天最高達約50道作業每天運行20小時以上Hadoop時代當前日志470GB/天當前366道作業平均67小時完成還有誰在用Hadoop?雅虎北京全球軟件研發中心中國移動研究院英特爾研究院金山軟件百度騰訊新浪搜狐IBMFacebookAmazonYahoo!Web ServersLog Collection ServersFilersData Warehousing on a ClusterOracle RACFederated MySQLWeb站點的典型Hadoop架構HadoopRich ClientMetaStore ServerMysqlSchedulerThrift ServerWebJobClientCLI/GUIClientProgramWeb Server淘寶Hadoop與Hive的使用標準輸出,標準出錯Web顯示(50030, 50060, 50070)NameNode,JobTracker, DataNode, TaskTracker日志本地重現: Local RunnerDistributedCache中放入調試代碼調試目的：查性能瓶頸，內存泄漏，線程死鎖等工具： jmap, jstat, hpr