Hadoop大數(shù)據(jù)項目開發(fā)——單詞統(tǒng)計功能代碼-提交任務(wù)實現(xiàn)目錄content02Reducer階段代碼編寫01Mapper階段代碼編寫03Drive階段代碼編寫提交任務(wù)代碼importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.mapreduce.Job;importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;提交任務(wù)代碼publicclassWordCountJob{ publicstaticvoidmain(String[]args)throwsException{ Jobjob=Job.getInstance(); job.setJarByClass(JobSubmitter.class);//1、封裝參數(shù)，jar包所在位置 //2、封裝參數(shù)，job調(diào)用的Mapper和Reducer實現(xiàn)類 job.setMapperClass(WordCountMapper.class); job.setReducerClass(WordCountReducer.class); //3、封裝參數(shù)，job調(diào)用的Mapper和Reducer實現(xiàn)類中key和value數(shù)據(jù)類型 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);

提交任務(wù)代碼 //4、封裝參數(shù)，處理的數(shù)據(jù)和最終結(jié)果數(shù)據(jù)路徑 FileInputFormat.setInputPaths(job,newPath("G:\\wordcount\\input"));//G:\\wordcount\\input為文件所存放的路徑 FileOutputFormat.setOutputPath(job,newPath("G:\\wordcount\\output"));//G:\\wordcount\\output為文件中單詞出現(xiàn)的次數(shù)。 //5、可以設(shè)置啟動的reduce數(shù)量 job.setNumReduceTasks(2); //6、提交job給YARN booleanres=job.waitForCompletion(true); System.exit(res?0:-1); }}最終結(jié)果課外拓展在上面代碼中，實現(xiàn)了本地測試功能，處理的數(shù)據(jù)來源本地，但是實際生產(chǎn)環(huán)境中，數(shù)據(jù)位于HDFS上，那么該如何解決？本地直接提交給集群運行生成Jar包提交給YARN運行課外拓展System.setProperty("HADOOP_USER_NAME","hadoop");Configurationconf=newConfiguration();conf.set("fs.defaultFS","hdfs://hdp-01:9000");Jobjob=Job.getInstance(conf);FileInputFormat.setInputPaths(job,newPath("/wordcount/input"));FileOutputFormat.setOutputPath(job,newPath("/wordcount/output"));本地直接提交給集群