深度搜索论文：面向电子商务网站的深度搜索与信息抽取研究.doc

深度搜索论文：面向电子商务网站的深度搜索与信息抽取研究【中文摘要】随着网络技术和数据库技术的飞速发展,互联网已成为海量信息的载体,如何从海量信息中准确、快速地抽取出有价值的信息已成为人们迫切需要解决的一大难题。电子商务网站具有动态Web页面、页面结构化程度高以及信息数量巨大且内容丰富等特点。实际上,电子商务网站属于Deep Web,Deep Web指的是存储在数据库中、不能通过静态超链接访问而需要通过动态网页技术访问的信息,用户需要向特定的查询接口提交查询请求后才能访问数据库中的数据。深度搜索就是针对Deep Web进行搜索。Deep Web不仅信息量大,而且具有面向某一领域、结构化程度较高且可以公开访问等特点。本文选择电子商务网站作为深度搜索的对象,研究商品信息的抽取。电子商务网站的站内搜索引擎为研究人员对网站进行深度搜索提供了便利,它可以根据用户输入的关键词,将与之相关的数据库内容以网页的形式动态呈现出来。我们可以利用这类查询接口进行深度搜索,通过模拟关键词的填充和提交,来获取我们需要的页面。通过深度搜索所获得的页面内含有大量商品信息,我们以此为信息源进行信息抽取。信息抽取研究中最关键的技术在于如何生成抽取规则,生成的方法有很多,但从生成的手段来说可以分为两种,一种是自动生成抽取规则,另一种是人工编写抽取规则,两种方法各有优缺点,有着各自的适用范围。自动生成的方法有利于针对不同结构的网站,但准确率较低;而人工编写的方法虽然需要人工参与,但抽取结果较为精确。本文针对电子商务网站进行信息抽取,抽取商品的各种信息,主要包括商品名称、价格、运费、商品相关信息等内容。由于同一网站下的子页面结构大体相同,因此,为了获得更为精确的抽取结果,我们采用了人工编写抽取规则的方法。本文的主要内容及创新点如下:1、设计一个关键词库接口,使得系统可以接受关键词库文件(文本文件,关键词之间以回车符隔开),将关键词库文件内的关键词一一导入至系统中,用于表单的自动填充和提交。另外,考虑到了效率问题,让系统在导入新关键词库文件时与旧关键词库进行对比,判断出新关键词,最后只接受更新的那一部分关键词,即达到关键词增量更新的。2、提取电子商务网站的HTML源码,通过对HTML源码进行分析,提取出网站中包含查询表单的那部分HTML源码。根据这部分HTML源码,利用WebBrowser控件方法对这些查询表单自动模拟关键词的填充和提交,用以获得每一个关键词所对应的初始页面。3、选择性的提取链接,只提取初始页面中的商品信息链接,而不提取广告和友情链接。另外,由于商品信息链接有多页,因此还需要通过获取“下一页”链接来得到更全的商品信息链接。本文介绍了多种获取“下一页”链接的方法,并提出了一个通用性较好的方法。4、通过正则表达式来定义抽取规则,根据不同的页面结构编写相应的正则表达式,从而进行信息抽取。抽取的结果以文本文件的格式保存,便于对关键词库文件进行更新。【英文摘要】With the rapid development of network and database technology, Internet has become a carrier of a great deal of information, how to extract the meaningful information accurately and rapidly from the mass of web information has become an urgent problem which people want to solve. There is a lot of web information in the E-commerce websites, these websitesweb pages are dynamic, and have high degree of structures. Actually, the E-commerce websites belongs to Deep Web, Deep Web is the information which can not access with hyperlinks and needs the dynamic web technology to access, users need to submit queries on the specific interface(the query interface) to access the data in the database. Deep search is the kind of search which aim at Deep Web. Deep Web has lots of information, high degree of structures and public access interface. So we choose the E-commerce websites to do the object of research of deep search, and get the purpose that extract the productsinformation. The search engines of E-commerce websites offer convenient for researchers to extracting information. Depending on some keywords that users input, these query interfaces can show users the information about keywords by web database dynamically. We can use these query interfaces to do some research into deep search, getting the web pages what we need by simulating the filling and submission of keywords.The webpages what get by deep search have a lot of productsinformation, we use these information source to extraction. The key point of web information extraction is how to generate extraction rules. There are two approaches to generating extraction rules, one is generating automatically, and another one is generating by manual work, these two approaches are all have advantages and shortcomings, they have their respective applicable scopes. The automatic approach is suitable for websites which have different structrue, but the precision rate of this approach is lower. Although the manual approach needs someone to write the regule rules, the precision rate of extraction results is higer. Due to the web pagesstructures of E-commerce websites are mainly the same, and we want to extract the information includes: productsname, productsprice, freight and other information about products, we choose the manual method what has more precise extraction results.The main works and innovations of this paper are as follows: 1、Design an interface of keywordsfiles, allowing the system to accept to the keywordsfiles(text files, there is a carriage return between two keywords), and putting these keywords in our system for filling and submitting query forms. Moreover, we consider the question of incremental keywords, the system of this paper do not accept the keywords what in the old keywordslibrary.2、Extracting the HTML codes of E-commerce websites. Extracting the part of query formsHTML codes of E-commerce websites by analyzing the HTML codes of E-commerce websites. According to these HTML codes, using WebBrowser Control to simulate the filling and submission of keywords, and getting the initial pages about keywords.3、Extracting the hyperlinks with selectivity, only extract the hyperlinks of productsinformation, but not the hyperlinks of advertisement and any other unrelated hyperlinks. Moreover, we need get more comprehensive hyperlinks of productsinformation by getting the hyperlinks of“next-page”, because of the multi-page hyperlinks of productsinformation. In this paper, we introduce some approaches of getting the hyperlinks of“next-page”, and propose an approach that has good applicability.4、According to the structures of different websites, generating the extraction rules by regular expression, thus information extraction. The extraction results are saved as text files which is convenient for updating keywordsfiles.【关键词】深度搜索 Deep Web Web信息抽取 URL提取 正则表达式【英文关键词】Deep Search Deep Web Web Information Extraction URL Collection Regular expression【目录】面向电子商务网站的深度搜索与信息抽取研究摘要3-5Abstract5-6第1章 绪论9-121.1 课题的研究背景及意义9-101.2 本文主要内容10-111.3 论文组织结构11-12第2章 深度搜索相关研究综述12-192.1 搜索引擎研究综述12-152.1.1 搜索引擎的概念122.1.2 搜索引擎的工作原理12-132.1.3 搜索引擎的分类13-152.2 深度搜索研究综述15-182.2.1 Deep Web 概述15-162.2.2 深度搜索的方法16-172.2.3 深度搜索的工作原理17-182.3 本章小结18-19第3章 Web 信息抽取研究综述19-293.1 Web 信息抽取的概念193.2 Web 信息抽取研究发展现状19-203.3 Web 信息抽取的方法20-233.3.1 Web 信息源20-213.3.2 结构化数据提取213.3.3 半结构化数据提取21-233.3.4 非结构