1、“天網”高性能分布式檢索系統的設計與實現趙江華（碩士論文，2002）摘 要隨著社會信息化的飛速發展，信息資源呈現爆炸式增長，信息系統的規模和復雜程度在不斷擴大，人們必須能夠存儲、處理和使用各種海量的數字化信息。信息檢索技術可以實現按內容檢索結構化和非結構化數據，被認為是解決信息過剩的有效途徑，有著極其重要的意義。搜索引擎和數字圖書館作為信息檢索最重要的兩個應用，已經產生了巨大社會的效益，其影響日益廣泛。本文以“天網”搜索引擎為應用背景，首先介紹信息檢索系統的各種基本問題，然后深入討論在單機系統中與系統性能相關的幾個因素和提高檢索效率的策略，試圖用定量的方法描述檢索系統中計算機硬件、數據規模等因

2、素和性能之間的關系。為了克服單機系統所固有的限制，我們設計和實現了“天網”的分布式檢索系統版本，保證了“天網”在維護上千萬網頁信息的情況下仍然能夠提供秒級的查詢響應，文中對此做了詳細介紹。進一步的系統模擬試驗表明：分布式情況下系統的瓶頸不是網絡帶寬，而是每個節點的檢索并發性能。進而，隨著“天網”系統并行化的完成，參與運行的機器數量不斷增長，我們感到對它本身的管理成為制約其進一步發展的重要因素。系統管理問題在計算機技術中存在已久，它在經濟和技術上的重要性逐漸被人們所認識到。由于信息技術在社會中的深入應用和普及，加之硬件和軟件愈來愈復雜，可靠性和可維護性都難以保證，系統管理逐漸成為制約信息系統發展

3、的瓶頸。本文從傳統的系統管理基本問題出發，結合網格計算和自動計算的方法和概念，提出了一種整體的解決方案，并且根據“天網”系統管理的需求，對今后發展給出了兩個探索性研究建議。關鍵詞：信息檢索，搜索引擎，倒排文件，分布式系統，并行，系統管理，網格計算，自動計算AbstractWith high-speed advances in information spaces, information resources are increasing explosively along with the scale and complexity of the information systems. The

4、re is a great need for storing, processing and implementing various aspects of digital information. Unlike common database technology, Information Retrieval (IR) can assist users in finding useful information from structured and unstructured data, which is considered an effective way of preventing i

5、nformation overload. Two important applications of IR, Search Engine and Digital Library, have produced many benefits to society and they are wielding an ever-increasing influence.The fundamental problems of an Information Retrieval System will be introduced first. “WebGather” Search Engine will be

6、utilized as a background for application in this thesis. Then, we will detail some factors relevant to system performance (such as computer hardware, data scales, etc) and methods of improving retrieval efficiency, trying to quantitatively describe the relations between these factors and performance

7、. In order to overcome the built-in limitation of a single computer system, we will design and implement a distributed retrieval system of “WebGather” that retrieves query responses in less than a second while indexing tens of millions of web pages. A system evaluation of it shows that to this distr

8、ibuted system the performance bottlenecks do not lie in network bandwidth but in the concurrent retrieval performance of the indexer. Furthermore, after the completion of a parallel system of “WebGather”, the quantities of computers in operation will be ever increasing, and we believe that the diffi

9、culty of managing the computers will prohibit the further development of “WebGather”. System management takes up a great deal of time in computer technology, the importance of this from the point of view of the economy and technological fields are gradually coming to light. Because of the extensive

10、application of information technology in society and the ongoing complexity of hardware and software without good reliability and maintainability, system management becomes the bottleneck of information system development. Starting from fundamental problems of traditional system administration, this

11、 thesis combines the methods and concepts of grid computing and autonomic computing to bring up a holistic solution. In addition, according to the demand on system management of “WebGather”, we will give two exploratory research proposals for the future.Keywords:Information retrieval, search engine, inverted file, distributed system, parallel, system management, grid computing, autonomic computing目錄第一節 序言1第二節 基本問題.32.1 倒排文件.32.2 相關度評價.52.3 系統運行模型.62.4 應用案例.9第三節 單機系統檢索性能分析123.1 通用計算性能分析.123.2