chapter01-part3-webmining_网络挖掘概述.ppt

1、2020年8月12日,网络挖掘综述,朱廷劭（Zhu, Tingshao）,Web Mining,Web mining - data mining techniques to automatically discover and extract information from Web documents/services (Etzioni, 1996). Web mining research integrate research from several research communities (Kosala and Blockeel, July 2000) such as: Databa

2、se (DB) Information retrieval (IR) The sub-areas of machine learning (ML) Natural language processing (NLP),Web Mining : Subtasks,Resource Finding Task of retrieving intended web-documents Information Selection Can identify information within given web pages (Etzioni et.al. 1997):Uses heuristics to

3、distinguish personal home pages from other web pages ShopBot (Etzioni et.al. 1997): Looks for product prices within web pages,Web Mining,Web Structure Mining,Web Content Mining,Web Page Content Mining,Search Result Mining,Web Usage Mining,General Access Pattern Tracking,Customized Usage Tracking,Web

4、 Mining Taxonomy,Search Engine Result Summarization Clustering Search Result (Leouski and Croft, 1996, Zamir and Etzioni, 1997): Categorizes documents using phrases in titles and snippets,Web Mining,Web Structure Mining,Web Content Mining,Web Page Content Mining,Search Result Mining,Web Usage Mining

5、,General Access Pattern Tracking,Customized Usage Tracking,Web Mining Taxonomy,Using Links PageRank (Brin et al., 1998) CLEVER (Chakrabarti et al., 1998) Using Generalization MLDB (1994) Uses a multi-level database representation of the Web. Counters (popularity) and link lists are used for capturin

6、g structure.,Web Mining,Web Structure Mining,Web Content Mining,Web Page Content Mining,Search Result Mining,Web Usage Mining,General Access Pattern Tracking,Customized Usage Tracking,Web Mining Taxonomy,Web Log Mining (Zaane, Xin and Han, 1998) Uses KDD techniques to understand general access patte

7、rns and trends. Can shed light on better structure and grouping of resource providers.,Web Mining,Web Structure Mining,Web Content Mining,Web Page Content Mining,Search Result Mining,Web Usage Mining,General Access Pattern Tracking,Customized Usage Tracking,Web Mining Taxonomy,Adaptive Sites (Perkow

8、itz and Etzioni, 1997) Analyzes access patterns of each user at a time. Web site restructures itself automatically by learning from user access patterns.,Web Content Mining,Discovery of useful information from web contents / data / documents Web data contents: text, image, audio, video, metadata and

9、 hyperlinks. Information Retrieval View ( Structured + Semi-Structured) Assist / Improve information finding Filtering Information to users on user profiles Database View Model Data on the web Integrate them for more sophisticated queries,Web Content Mining,Agent-based Approaches: Intelligent Search

10、 Agents Information Filtering/Categorization Personalized Web Agents Database Approaches: Multilevel Databases Web Query Systems,Intelligent Search Agents,Locating documents and services on the Web: WebCrawler, Alta Vista (): scan millions of Web documents and create index of words (too many irrelev

11、ant, outdated responses) MetaCrawler: mines robot-created indices Retrieve product information from a variety of vendor sites using only general information about the product domain: ShopBot,Intelligent Search Agents (Contd),Rely either on pre-specified domain information about particular types of d

12、ocuments, or on hard coded models of the information sources to retrieve and interpret documents: Harvest FAQ-Finder Information Manifold OCCAM Parasite Learn models of various information sources and translates these into its own concept hierarchy: ILA (Internet Learning Agent),Information Filterin

13、g/Categorization,Using various information retrieval techniques and characteristics of open hypertext Web documents to automatically retrieve, filter, and categorize them. HyPursuit: uses semantic information embedded in link structures and document content to create cluster hierarchies of hypertext

14、 documents, and structure an information space BO (Bookmark Organizer): combines hierarchical clustering techniques and user interaction to organize a collection of Web documents based on conceptual information,Personalized Web Agents,This category of Web agents learn user preferences and discover W

15、eb information sources based on these preferences, and those of other individuals with similar interests (using collaborative filtering) WebWatcher PAINT Syskill&Webert GroupLens Firefly others,Web Structure Mining,To discover the link structure of the hyperlinks at the inter-document level to gener

16、ate structural summary about the Website and Web page. Direction 1: based on the hyperlinks, categorizing the Web pages and generated information. Direction 2: discovering the structure of Web document itself. Direction 3: discovering the nature of the hierarchy or network of hyperlinks in the Websi

17、te of a particular domain.,Web Structure Mining,Finding authoritative Web pages Retrieving pages that are not only relevant, but also of high quality, or authoritative on the topic Hyperlinks can infer the notion of authority The Web consists not only of pages, but also of hyperlinks pointing from o

18、ne page to another These hyperlinks contain an enormous amount of latent human annotation A hyperlink pointing to another Web page, this can be considered as the authors endorsement of the other page,Web Usage Mining,Web usage mining also known as Web log mining mining techniques to discover interes

19、ting usage patterns from the secondary data derived from the interactions of the users while surfing the web,Web Logs,Web servers have the ability to log all requests Web server log formats: Most use the Common Log Format (CLF) New, Extended Log Format allows configuration of log file Generate vast

20、amounts of data,Potential Data Sources,Client Computer,Client-level logs,User Behaviors,Modem,ISP Server,Proxy-level logs,Web Server,Server-level logs,Content Server,Content-level logs,Site Content,The Web Usage Mining Process,Content and Structure Data,Preprocessing,Pattern Discovery,Pattern Analys

21、is,Usage Data,Preprocessed Clickstream Data,Rule, Patterns and Statistics,Interesting Rule, Patterns and Statistics,Problems with Web Logs,Identifying users Clients may have multiple streams Clients may access web from multiple hosts Proxy servers: many clients/one address Proxy servers: one client/

22、many addresses Data not in log POST data (i.e., CGI request) not recorded Cookie data stored elsewhere,Cont,Missing data Pages may be cached Referring page requires client cooperation When does a session end? Use of forward and backward pointers Typically a 30 minute timeout is used Web content may be dynamic May not be able to reconstruct what the user saw Use of spiders and automated agents automatic request we pages,Data Preprocessing,Like most data mining tasks, web log mining requires preproce