1、安裝說明 npm install express -gnpm install cheerionpm install requestnpm install qsnpm config set prefix "f:notejsapp"npm config set cache "f:notejsappcache"npm install -g express-generatornpm install mysqlnpm install tcp-pingnpm install httpnpm install fsnpm install n-libmkdir app c

2、d appnpm init一直回車，設置默認參數 數據表(mysql)CREATE TABLE robot_tasks ( url varchar(200) DEFAULT NULL, site varchar(50) DEFAULT NULL, status int(11) DEFAULT '0', lastupdate timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, count int(10) unsigned DEFAULT '1', rand_id int

3、(11) DEFAULT NULL, seqid bigint(20) DEFAULT NULL, UNIQUE KEY url (url) ENGINE=InnoDB DEFAULT CHARSET=utf8;獲取免費代理IP，將運行結果復制到后面的代理IP數組-task_getproxy.js start-var request = require('request'),cheerio = require('cheerio'),mysql = require('mysql'), tcpp= require('tcp-ping'

4、),query= require("./mysqludf.js"), /mysql連接池proxys = ; / 代理ip配置 mypool = ; / 任務池proxysites = siteurl : 'regx_row : '#ip_list tr',tdid_ip : 1,trypages : 5,siteurl : 'regx_row : '',trypages : 5,siteurl :'regx_row : '.table-striped tbody tr',tdid_ip : 0,try

5、pages : 10,processcount = 0,Queue_Is_Started = false, /異步判斷隊列是否已開始執行Error_count = 0,Queue_Per_COunt = 50; /主任務執行狀態;var myurl = '' /異步函數 保存抓取的代理數據 function addproxy(ipinfo)be(ipinfo.ip, ipinfo.port, function(err, available) if(available)proxys.push(ip: ipinfo.ip, port: ipinfo.port);if

6、(!Queue_Is_Started && proxys.length >= 80)/ 只獲取80個 展示出來 msg('開始處理任務隊列', 1);do_Task_Queue(0);/開始處理任務隊列Queue_Is_Started = true;);/獲取代理服務器 依次從備選的代理站點抓取function getproxys(n)if(n > proxysites.length - 1)msg('抓取代理失敗-n='+n, 2);return;msg('開始抓取代理IP數據'+n+'', 1);p

7、rocesscount = 0;Error_count = 0;proxysite = proxysitesn;for(var i = 1; i <= proxysite.trypages; i+)var pageurl = proxysite.siteurl.replace('pageid',i);request(url : pageurl,timeout: 10000,method : 'GET',headers: 'User-Agent':'Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

8、20100101 Firefox/5.0',function(error, response, body) processcount+;if(error)Error_count+;msg('獲取http代理失敗'+Error_count+''+error.code, 3);else/console.log(body);$ = cheerio.load(body);if(proxysite.regx_row !='')$(proxysite.regx_row).each(function()var tds = $(this).find(&#

9、39;td');var myip = getIp($(tdsproxysite.tdid_ip).text();var myport = $(tdsproxysite.tdid_ip+1).text();/console.log(myport);/return;if(myip!='')addproxy(ip: myip, port: myport););if(pageurl.indexOf('')> 0)var table = $('.list_title').parent().parent();$(table).find('

10、;tr').each(function()var myip = getIp($(this).find('script').text();var myport = $(this).find('script').parent().next().text();if(myip!='')addproxy(ip: myip, port: myport););if(proxys.length <=0 && processcount > 1 && Error_count = 2)getproxys(n+1);)

11、;/主調度函數function dostart()/抓取代理服務器列表 step1getproxys(0);/ 消息輸出function msg(str, level)console.log(str); /執行任務隊列 function do_Task_Queue(task_Queue_No) console.log(proxys);/ 隨機選取一個代理IPfunction GetProxy() var randomNum = parseInt(Math.floor(Math.random() * proxys.length); var proxy = proxysrandomNum;/ret

12、urn '55:9999' return 'http:/' + proxy.ip + ':' + proxy.port;/取得字符串的數值function getIp(str)result= (str+'').match(/d+/g);/console.log(result);if(result = null)return ''if(result.length = 4)return result0+'.'+result1+'.'+result2+&#

13、39;.'+result3;return result;/取得字符串的數值 function getNum(str)result= (str+'').match(/d+(.d+)?/g);if(result = null)return 0;return result0; /獲取頁面中的json字符串function getjsonstr(body, jsonname)var i = body.indexOf(jsonname);var str = body.substr(i, body.indexOf('', i) - i );var i = str.i

14、ndexOf('');var str = str.substr(i, str.length - i);return str.replace(/"/g,'').replace(/'/g,'').split(',');/開始抓取數據dostart();-task_getproxy.js end-task_geturls.js start-var request = require('request'), cheerio = require('cheerio'),http= requir

15、e('http'),mysql = require('mysql'),zlib = require('zlib'),fs = require('fs');/調試消息輸出function msg(str)console.log(str); / 獲取考拉網的站點地圖function getkaolaurls()request(url : 'timeout: 10000,method : 'GET',function(error, response, body) if(error)msg('>>

16、;抓取考拉站點地圖主文件出錯-'+error.code);return;msg('>>抓取考拉站點地圖主文件完成');var $ = cheerio.load(body, ignoreWhitespace:true);/, xmlMode:true var submaps = $('url');var count = 0;for(var i in submaps)submap = submapsi;if( = 'url')savekaolaxmlfile($(submapsi).find('loc

17、').text(), count);count = count + 1;); /考拉網地圖文件分析function savekaolaxmlfile(xmlurl, count)var tmp = xmlurl.split('/');var d = new Date();var daystr = d.getFullYear() + '-' + d.getMonth()+ '-' + d.getDate();var filename = './download/' + 'KaoLa_' + daystr +

18、'_' +tmptmp.length -1 + '.xml' var output = fs.createWriteStream(filename);request(xmlurl).pipe(zlib.createGunzip().pipe(output);setTimeout(function() msg('>>>>抓取考拉網子地圖文件:'+xmlurl);var xmlcontent = fs.readFileSync(filename,'utf-8');savetodb(filename, xmlco

19、ntent, '考拉網');, count*2000);/將地圖文件保存到數據庫function savetodb(filename, xmlcontent, sitename)var $ = cheerio.load(xmlcontent);var urls = $('url');var conn = mysql.createConnection(host: 'localhost',user: 'root',password: '',database:'test',port: 3306);conn

20、.connect();for(var i = 0; i< urls.length; i+)var url = $(urlsi).find('loc').text();if(url.indexOf('/product') > 0)sql = "insert into robot_tasks(url,site,rand_id) values('"+url.replace(/'/g,'')+"','"+sitename+"',"+0+&quo

21、t;) ON DUPLICATE KEY UPDATE count = count + 1"conn.query(sql, function(err, rows, fields) if (err) throw err;/console.log('The solution is: ', rows0.TABLE_NAME););conn.end();msg('>>>>>>'+filename+'.保存記錄數: '+urls.length);function getproxyIP() / 通過task_ge

22、tproxy.js獲取的代表IP列表 var proxyips = ip: '53', port: '80' , ip: '6', port: '80' , ip: '76', port: '8123' , ip: '95', port: '8123' , ip: '28', port: '8123' , ip: '11

23、3', port: '8080' , ip: '36', port: '8123' , ip: '9', port: '8123' , ip: '83', port: '80' , ip: '11', port: '8123' , ip: '52', port: '8123' , i

24、p: '8', port: '9000' , ip: '7', port: '80' , ip: '1', port: '80' , ip: '39', port: '8123' , ip: '42', port: '8088' , ip: '0', port: '80' , ip: '05', port: '8123' ,