1、-實(shí)驗(yàn)報(bào)告2021 / 2021 學(xué)年第二學(xué)期課程名稱(chēng)云計(jì)算實(shí)驗(yàn)名稱(chēng)基于Hadoop的數(shù)據(jù)分析實(shí)驗(yàn)時(shí)間2021 年5月20日指導(dǎo)單位指導(dǎo)教師學(xué)生班級(jí)*學(xué)院(系)專(zhuān) 業(yè). z-. z-實(shí) 驗(yàn) 報(bào) 告實(shí)驗(yàn)名稱(chēng)指導(dǎo)教師實(shí)驗(yàn)類(lèi)型上機(jī)實(shí)驗(yàn)學(xué)時(shí)2實(shí)驗(yàn)時(shí)間2021 .5.20一、 實(shí)驗(yàn)?zāi)康暮鸵蟆緦?shí)驗(yàn)要求】掌握基于Hadoop單機(jī)版的數(shù)據(jù)分析【實(shí)驗(yàn)?zāi)康摹繉W(xué)會(huì)利用Hadoop對(duì)不同數(shù)據(jù)集的分析與挖掘；二、實(shí)驗(yàn)環(huán)境虛擬機(jī)Ubuntu14.04，hadoop-1.2.1，eclipse三、實(shí)驗(yàn)原理及容(1)在第一次實(shí)驗(yàn)的根底上安裝eclipse1.將eclipse解壓到usr/local目錄下：-C/usr/lo

2、cal2.授權(quán)：sudo chmod -R 777 /usr/local/eclipse3. 將hadoop插件拷貝到/usr/local/eclipse/plugins下4.設(shè)置eclipse的Java環(huán)境：在eclipse下創(chuàng)立一個(gè)jre文件夾sudo mkdir jre，進(jìn)入該文件夾：cd jre，參加一條連接指令ln -s /usr/lib/jvm/java-7-sun/bin bin(2)在eclipse中添加Map/Reduce：點(diǎn)擊上方window選項(xiàng)，依次選擇open perspective，other，Map、Reduce，如以下圖所示：(3)設(shè)置Map/Reduce loc

3、ation，選擇Map/Reduce locations，new hadoop location，將其中的容設(shè)置成以下圖所示的容：設(shè)置Advanced parameters中的tmp文件夾位置為/usr/local/hadoop/tmp，如以下圖所示：(4)設(shè)置hadoop的安裝路徑：依次點(diǎn)擊window，preferences，Hadoop Map/Reduce，設(shè)置hadoop安裝路徑為/usr/local/hadoop(5)在桌面新建三個(gè)文件，如以下圖：(6) 新建一個(gè)Map/Reduce Project：右擊左側(cè)空白處，選擇新建一個(gè)如以下圖所示的Map/Reduce Project：點(diǎn)

4、擊Ne*t，輸入工程名為average，再點(diǎn)擊finish。(7) 新建一個(gè)Java class：右擊左側(cè)的average，依次選擇New，class，設(shè)置容如以下圖所示：點(diǎn)擊finish完成。(8)參加Java代碼：package .hebut.mr;import java.io.IOE*ception;import java.util.Iterator;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.ap

5、ache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Te*t;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import 

6、org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.Te*tInputFormat;import ileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.Te*tOutputFormat;import org.apache.hadoop.util.GenericOptionsParser; public class 

7、;Score  public static class Map e*tends Mapper<LongWritable, Te*t, Te*t, IntWritable>  / 實(shí)現(xiàn)map函數(shù) public void map(LongWritable key, Te*t value, Conte*t conte*t) throws IOE*ception, InterruptedE*ception  / 將輸入的純文本文件的

8、數(shù)據(jù)轉(zhuǎn)化成String String line = value.toString(); / 將輸入的數(shù)據(jù)首先按行進(jìn)展分割StringTokenizer tokenizerArticle = new StringTokenizer(line, "n"); / 分別對(duì)每一行進(jìn)展處理 while (tokenizerArticle.hasMoreElements()  / 每行按空格劃分 StringTokenizertokenizerLine= newStri

9、ngTokenizer(tokenizerArticle.ne*tToken(); String strName = tokenizerLine.ne*tToken();/ 學(xué)生局部String strScore = tokenizerLine.ne*tToken();/ 成績(jī)局部Te*t name = new Te*t(strName);int scoreInt = Integer.parseInt(strScore);/ 輸出和成績(jī) conte*t.write(name, new IntWritable(sco

10、reInt);  public static class Reduce e*tends Reducer<Te*t, IntWritable, Te*t, IntWritable> / 實(shí)現(xiàn)reduce函數(shù) public void reduce(Te*t key, Iterable<IntWritable> values,Conte*t conte*t) throws IOE*ception, InterruptedE*ception int sum =

11、 0;int count = 0; Iterator<IntWritable> iterator = values.iterator(); while (iterator.hasNe*t() sum += iterator.ne*t().get();/ 計(jì)算總分 count+;/ 統(tǒng)計(jì)總的科目數(shù) int average = (int) sum / count;/ 計(jì)算平均conte*t.write(key, new IntWritable(average);   

12、           public static void main(String args) throws E*ception         Configuration conf = new Configuration();        / 這句話(huà)很關(guān)鍵 

13、0;      conf.set("mapred.job.tracker", "localhost:9001");  conf.set ("mapred.jar", "Score.jar");   String ioArgs = new String  "score_in", "score_out"   &#

14、160;     String otherArgs = new GenericOptionsParser(conf, ioArgs).getRemainingArgs();        if (otherArgs.length != 2)             System.err.println("Usage:

15、Score Average <in> <out>");            System.e*it(2);          Job job = new Job(conf, "Score Average");        job.setJarByCla

16、ss(Score.class)        / 設(shè)置Map、bine和Reduce處理類(lèi)        job.setMapperClass(Map.class);        job.setbinerClass(Reduce.class);        job.setReducerClass

17、(Reduce.class);  / 設(shè)置輸出類(lèi)型        job.setOutputKeyClass(Te*t.class);        job.setOutputValueClass(IntWritable.class);        / 將輸入的數(shù)據(jù)集分割成小數(shù)據(jù)塊splites，提供一個(gè)RecordReder的實(shí)現(xiàn)&#

18、160;       job.setInputFormatClass(Te*tInputFormat.class);        / 提供一個(gè)RecordWriter的實(shí)現(xiàn)，負(fù)責(zé)數(shù)據(jù)輸出        job.setOutputFormatClass(Te*tOutputFormat.class);      

19、60; / 設(shè)置輸入和輸出目錄        FileInputFormat.addInputPath(job, new Path(otherArgs0);        FileOutputFormat.setOutputPath(job, new Path(otherArgs1);        System.e*it(job.waitForpletion(true) ? 0 : 1);    (注意將conf.set("mapred.job.tracker