“天网”高性能分布式检索系统的设计与实现.doc

“天网”高性能分布式检索系统的设计与实现赵江华（硕士论文，2002）摘 要随着社会信息化的飞速发展，信息资源呈现爆炸式增长，信息系统的规模和复杂程度在不断扩大，人们必须能够存储、处理和使用各种海量的数字化信息。信息检索技术可以实现按内容检索结构化和非结构化数据，被认为是解决信息过剩的有效途径，有着极其重要的意义。搜索引擎和数字图书馆作为信息检索最重要的两个应用，已经产生了巨大社会的效益，其影响日益广泛。本文以“天网”搜索引擎为应用背景，首先介绍信息检索系统的各种基本问题，然后深入讨论在单机系统中与系统性能相关的几个因素和提高检索效率的策略，试图用定量的方法描述检索系统中计算机硬件、数据规模等因素和性能之间的关系。为了克服单机系统所固有的限制，我们设计和实现了“天网”的分布式检索系统版本，保证了“天网”在维护上千万网页信息的情况下仍然能够提供秒级的查询响应，文中对此做了详细介绍。进一步的系统模拟试验表明：分布式情况下系统的瓶颈不是网络带宽，而是每个节点的检索并发性能。进而，随着“天网”系统并行化的完成，参与运行的机器数量不断增长，我们感到对它本身的管理成为制约其进一步发展的重要因素。系统管理问题在计算机技术中存在已久，它在经济和技术上的重要性逐渐被人们所认识到。由于信息技术在社会中的深入应用和普及，加之硬件和软件愈来愈复杂，可靠性和可维护性都难以保证，系统管理逐渐成为制约信息系统发展的瓶颈。本文从传统的系统管理基本问题出发，结合网格计算和自动计算的方法和概念，提出了一种整体的解决方案，并且根据“天网”系统管理的需求，对今后发展给出了两个探索性研究建议。关键词：信息检索，搜索引擎，倒排文件，分布式系统，并行，系统管理，网格计算，自动计算abstractwith high-speed advances in information spaces, information resources are increasing explosively along with the scale and complexity of the information systems. there is a great need for storing, processing and implementing various aspects of digital information. unlike common database technology, information retrieval (ir) can assist users in finding useful information from structured and unstructured data, which is considered an effective way of preventing information overload. two important applications of ir, search engine and digital library, have produced many benefits to society and they are wielding an ever-increasing influence.the fundamental problems of an information retrieval system will be introduced first. “webgather” search engine will be utilized as a background for application in this thesis. then, we will detail some factors relevant to system performance (such as computer hardware, data scales, etc) and methods of improving retrieval efficiency, trying to quantitatively describe the relations between these factors and performance. in order to overcome the built-in limitation of a single computer system, we will design and implement a distributed retrieval system of “webgather” that retrieves query responses in less than a second while indexing tens of millions of web pages. a system evaluation of it shows that to this distributed system the performance bottlenecks do not lie in network bandwidth but in the concurrent retrieval performance of the indexer. furthermore, after the completion of a parallel system of “webgather”, the quantities of computers in operation will be ever increasing, and we believe that the difficulty of managing the computers will prohibit the further development of “webgather”. system management takes up a great deal of time in computer technology, the importance of this from the point of view of the economy and technological fields are gradually coming to light. because of the extensive application of information technology in society and the ongoing complexity of hardware and software without good reliability and maintainability, system management becomes the bottleneck of information system development. starting from fundamental problems of traditional system administration, this thesis combines the methods and concepts of grid computing and autonomic computing to bring up a holistic solution. in addition, according to the demand on system management of “webgather”, we will give two exploratory research proposals for the future.keywords:information retrieval, search engine, inverted file, distributed system, parallel, system management, grid computing, autonomic computing目录第一节 序言1第二节 基本问题.32.1 倒排文件.32.2 相关度评价.52.3 系统运行模型.62.4 应用案例.9第三节 单机系统检索性能分析123.1 通用计算性能分析.123.2