数据挖掘-1-商务智能与数据挖掘概述

1、数据挖掘理论与应用哈尔滨工业大学 管理学院叶强yeqiangSAP成立于1972年，总部位于德国沃尔多夫市是全球最大的企业管理软件供应商、全球最大ERP软件商，全球第三大独立软件供应商Fortune Global 500 企业80%是SAP的用户。ORACLE公司成立于1977年，总部位于美国加州 ，全球第二大独立软件供应商和第二大ERP软件供应商，1998年Oracle宣布大举进军应用软件市场 Whats the same ？Business IntelligenceIDC的一项研究表明北美企业纷纷转向商务智能工具的开发和应用。商业智能软件市场方兴未艾，为低迷的IT产业带来一束曙光。根据弗瑞

2、斯特研究中心（Forrester Research）的调查发现，去年底有44的企业表示，计划在今年购买BI软件。另外，据IDC提供的数据，在五年之内，商业智能市场规模将从现在的55亿美元扩大到157亿美元。正是看到了BI市场的巨大“钱途”，一些IT服务提供商纷纷推出自己的BI软件，如Oracle、微软、IBM公司、国内的金碟公司等均大力开发和推广BI软件。与此同时，一些商业企业、制造企业也在积极应用BI软件，为其决策提供有价值的服务。研究生选题美孚公司的客户分析美孚石油公司毕马威 KPMG参考书：Jiawei Han, Micheline Kambr. 数据挖掘概念与技术（影印版）.高等教育出

3、版社,2001.5(英文)韩家炜 , Micheline Kambr. 数据挖掘概念与技术.机械工业出版社,2001.8（中译本）Margaret H. Dunham. 数据挖掘教程. 清华大学出版社。2003(英文)史忠植. 知识发现. 清华大学出版社, 2002.陈文伟，黄金才. 数据仓库与数据挖掘 .人民邮电出版社，2004 Olivia Parr Rud. 数据挖掘实践. 机械工业出版社. 2003邢文训.现代优化计算方法.清华大学出版社,1999.阎平凡、张长水. 人工神经网络与模拟进化计算. 清华大学出版社,2000刘勇、康立山、陈毓屏. 非数值并行算法遗传算法. 科学出版社, 2

4、000MISQ (MIS Quarterly)ISR (Information System Research)CACM (Communication of the ACM)MS (Management Science)JMIS (Journal of Management Information Systems)AI (Artificial Intelligence)DSI (Decision Science)HBR (Harvard Business Review)IEEETrans (IEEE Transactions )AIMag (AI Magazine)EJIS (European

5、 Journal of Information Systems)DSS (Decision Support Systems)Ranking of MIS JournalsJournals of Data Mining Data Mining and Knowledge Discovery (DMKD, since 1997) SCI （2.8） Springer Machine learning (SCI 3.258) Monthly Springer IEEE Transactions on Knowledge and Data Engineering (TKDE) SCI (1.243)K

6、nowledge and Information Systems (KAIS, since 1999) SCI Springer Many others, DSS、ISR DATA & KNOWLEDGE ENGINEERING ACM Transaction on Information Systems ACM Transactions on Database Systems ACM Transactions on Knowledge Discovery from Data (TKDD) 国际会议SIGKDD (International Conference on Knowledge Di

7、scovery and Data Mining ,ACM SIGKDD , 2006 August 23-26 ,Philadelphia）SIGMOD (Special Interest Group on Management Of Data, ACM SIGMOD, Association for Computing Machinery, June 2006, Chicago )ICIS (International Conference on Information Systems, AIS)HICSS （Hawaii International Conference on System

8、 Science, IEEE）/在线直播Han Jiawei（韩家炜） 1 商务智能概述商务智能的含义 商务智能（Business Intelligence, 缩写为BI）指利用计算机及计算机网络，从商业数据存储中提取与分析企业关注信息的智能化的数据分析处理系统。 商务智能的提出九十年代末，美国商务智能的核心技术 数据仓库DW , 在线分析处理 OLAP, 数据挖掘DM BIBusiness Intelligence DW Data Warehousing OLAP On-Line Analytical Processing DM Data Mining Architecture of Bus

9、iness Intelligence System(Won Kim, 1998 2nd Worldwide Computing and its Applications)Business Intelligence ApplicationsData miningengineOLAPengineData Warehouse/Data MartBase dataBase dataBase data1）数据仓库 为满足管理决策中的数据需求，W.H.Inmon，在1992最先提出了数据仓库的概念。按照Inmon的定义，数据仓库是面向主题的、集成的、稳定的、不同时间的数据集合，用于支持经营管理的决策制定过

10、程。目前已经成为企业级决策系统的重要组成部分。 产品1产品2数据仓库的多维模型1季度地区维产品维时间维2季度3季度4季度关系型数据模型（二维）多维数据模型：星型 ：事实表（中心表），附属表（维表）雪花型星云型Example of Star Schema time_keydayday_of_the_weekmonthquarteryeartimelocation_keystreetcitystate_or_provincecountrylocationSales Fact Table time_key item_key branch_key location_key units_sold dol

11、lars_sold avg_salesMeasuresitem_keyitem_namebrandtypesupplier_typeitembranch_keybranch_namebranch_typebranchExample of Snowflake Schematime_keydayday_of_the_weekmonthquarteryeartimelocation_keystreetcity_keylocationSales Fact Table time_key item_key branch_key location_key units_sold dollars_sold av

12、g_salesMeasuresitem_keyitem_namebrandtypesupplier_keyitembranch_keybranch_namebranch_typebranchsupplier_keysupplier_typesuppliercity_keycitystate_or_provincecountrycityExample of Fact Constellationtime_keydayday_of_the_weekmonthquarteryeartimelocation_keystreetcityprovince_or_statecountrylocationSal

13、es Fact Tabletime_key item_key branch_key location_key units_sold dollars_sold avg_salesMeasuresitem_keyitem_namebrandtypesupplier_typeitembranch_keybranch_namebranch_typebranchShipping Fact Tabletime_key item_key shipper_key from_location to_location dollars_cost units_shippedshipper_keyshipper_nam

14、elocation_keyshipper_typeshipper2）OLAP 为满足基于大型数据库的复杂查询、决策分析等需求，弥补OLTP（On-Line Transaction Processing, 在线事务处理）在功能上的不足，90年代初出现了OLAP技术（E.F. Codd，1993），目前OLAP 已经成为大部分数据仓库的重要分析工具。 多维数据模型上的OLAP操作上卷（roll-up)下钻（drill-down)切片（dice)切块（slice)旋转（rotate),旋轴（pivot） 产品1产品2多维数据模型上的操作1季度地区维产品维时间维2季度3季度4季度3）数据挖掘 查询驱动

15、的OLAP可以按要求将数据展示在决策者面前，却无法自动发现潜藏在数据中的有用信息，大大降低了数据的使用价值。为实现对潜藏信息的自动发掘，90年代中期，出现了数据挖掘技术（Data Mining）。数据挖掘是源于KDD（Knowledge Discovery in Database,数据库中的知识发现）的一项以人工智能为基础的数据分析技术，其主要功能是在大量数据中自动发现潜在有用的知识，这些知识可以被表示为概念、规则、规律、模式等。2数据挖掘概述What Motivated Data Mining ?The major reason that data mining has attracted

16、a great deal of attention in the information industry in recent years is due to the wide availability of huge amounts of data and the imminent need for turning such data into useful information and knowledge.尿布与啤酒的故事全球连锁零售巨头沃尔玛公司公司Teradata（数据仓库）事业部，提供数据仓库系统Data Collection and Database Creation (1960

17、s and earlier)-primitive file processingDatabase Management Systems (1970-1980s)-Hierarchical and network database systems-Relational database systems-Indexing and data organization techniques: B+tree, hashing tree-Query language: SQL, etc.-OLTPAdvanced Database Systems(mid-1980spresent)-Object Orie

18、nted-object-relational-Application-oriented: spatial, temporalWeb-based Databases Systems(1990s-present)-XML-based database systems-web miningData Warehousing and Data Mining (late 1980s-present)-data warehouse and OLAP technology -Data mining and knowledge discoveryNew Generation of Integrated Info

19、rmation Systems (2000-)The evolution of database technologyWhats Data Mining ?Data Mining refers to extracting or “mining” Knowledge from large amount of data.Handetal.(2000)：“Data mining is the process of seeking interesting or valuable information in large data bases.（数据挖掘是一种在大型数据库中寻找你感兴趣或是有价值信息的过

20、程。）” Data Mining and KDDData mining is a step of KDD ?Data mining is KDD ?Data Mining: A KDD ProcessData mining:core of knowledge discovery processData CleaningData IntegrationDatabasesData WarehouseKnowledgeTask-relevant DataSelectionData MiningPattern EvaluationData Mining and Business Intelligenc

21、e Increasing potentialto supportbusiness decisionsDecision makerBusiness Analyst DataAnalystDBA MakingDecisionsData PresentationVisualization TechniquesData MiningInformation DiscoveryData ExplorationOLAPStatistical Analysis, Querying and ReportingData Warehouses / Data MartsData Sources Files, Info

22、rmation Providers, Database Systems, OLTPData mining is a confluence of multiple disciplinesDatabase technologyStatisticsMachine learningOther disciplinesVisualizationData MiningInformationscienceClassification of Data Mining SystemsCriteria:The kinds of databases mined The kinds of knowledge minedT

23、he kinds of techniques utilizedThe applications adaptedClassification according to the kinds of database minedData models:Relational Data MiningObject-oriented Data MiningObject-relational Data MiningData warehouse Data MiningTypes of data:Spatial Data MiningTime-series Data MiningText Data MiningMu

24、ltimedia Data MiningWeb Data MiningClassification according to the kinds of knowledge minedDM for characterization knowledgeDM for discrimination knowledgeDM for association knowledgeDM for classification knowledgeDM for clustering knowledgeDM for outlier analysis DM for evolution analysisClassifica

25、tion according to the kinds of techniques utilizedMachine learningStatisticsVisualizationPattern recognitionNeural networksGenetic AlgorithmClassification according to the applications adaptedFinanceSales data miningTelecommunicationsDNA sequence analysisStock markets analysisE-mail processing Data

26、preprocessingData cleaningMissing values; Noisy Data; Inconsistent dataData integration and transformationThe merging of data from multiple data storesTransform into the forms appropriate for miningData reductionData cube aggregation; dimensionality reduction; data compression; numerosity reductionD

27、iscretization and concept Hierarchy Generation allComputer accessoryprintersoftwarecomputerWrist padmouseb/wcolorapplicationOSlaptopdesktopIBMHPMicrosoftToshibaA concept hierarchy Pattern evaluation Interesting MeasuresSimplicityRule length (conjunction normal form)Certainty Confidence Utility SupportNoveltyPresentation of discovered patternsRulesTableCrosstabPie chartBar chartData cubeNeural networ