标准规范工具和发展趋势.ppt

数据挖掘,第八章：标准规范、工具和发展趋势,本章内容,8.1数据挖掘标准与规范8.2数据挖掘工具8.3数据挖掘的研究趋势,基本要求：了解数据挖掘在应用中的相关标准规范及未来的研究趋势。,8.1数据挖掘标准与规范,数据挖掘过程模型是确保数据挖掘工作顺利进行的关键。典型的过程模型有：SPSS的5A模型评估(Assess)、访问(Access)、分析(Analyze)、行动(Act)、自动化(Automate)SAS的SEMMA模型采样(Sample)、探索(Explore)、修正(Modify)、建模(Model)、评估(Assess)跨行业数据挖掘过程标准CRISP-DM(CrossIndustryStandardProcessforDataMining)。TwoCrows公司的数据挖掘过程模型，它与正在建立的CRISP-DM有许多相似之处。,数据挖掘相关标准CRISP-DM（交叉行业数据挖掘过程标准，CrossIndustryStandardProcessforDataMining）。SPSS、NCR以及DaimlerChrysler三个在数据挖掘领域经验丰富的公司发起建立一个社团，目的建立数据挖掘方法和过程的标准,8.1数据挖掘标准与规范,Crisp-DM,ProjectObjectives,DataUnderstanding,DataPreparation,Modeling,Evaluation,Reporting,BackgroundRequirements,assumptions,constraintsTerminologyDatamininggoals&successcriteriaProjectplan,InitialDatacollectionreportDatadescriptionreportDataExplorationreportDataqualityreport,DatadescriptionreportDatapre-processingsteps,ModelingassumptionTestdesignModeldescriptionModelassessment(inc.validation),Assessmentofdataminingresultswithrespecttoobjectives,Finalreport:Summary:ObjectivesDataMiningprocessDataMiningresultsDataMiningassessment-ConclusionsFuturework,(BusinessUnderstanding),(Deployment),WidelyacceptedPROCESSMODELfordataminingProvidesaframeworkfordescribingthemodelingprocessindetail“BESTPRACTICE”,BusinessUnderstandingPhaseUnderstandthebusinessobjectivesWhatisthestatusquo?UnderstandbusinessprocessesAssociatedcosts/painDefinethesuccesscriteriaDevelopaglossaryofterms:speakthelanguageCost/BenefitAnalysisCurrentSystemsAssessmentIdentifythekeyactorsMinimum:TheSponsorandtheKeyUserWhatformsshouldtheoutputtake?IntegrationofoutputwithexistingtechnologylandscapeUnderstandmarketnormsandstandards,8.1数据挖掘标准与规范,BusinessUnderstandingPhaseTaskDecompositionBreakdowntheobjectiveintosub-tasksMapsub-taskstodataminingproblemdefinitionsIdentifyConstraintsResourcesLawe.g.DataProtectionBuildaprojectplanListassumptionsandrisk(technical/financial/business/organisational)factors,8.1数据挖掘标准与规范,DataUnderstandingPhaseCollectDataWhatarethedatasources?InternalandExternalSources(e.g.Axiom,Experian)Documentreasonsforinclusion/exclusionsDependonadomainexpertAccessibilityissuesArethereissuesregardingdatadistributionacrossdifferentdatabases/legacysystemsWherearethedisconnects?,8.1数据挖掘标准与规范,DataUnderstandingPhaseDataDescriptionDocumentdataqualityissuesComputebasicstatisticsDataExplorationSimpleunivariatedataplots/distributionsInvestigateattributeinteractionsDataQualityIssuesMissingValues:UnderstanditssourceStrangeDistributions,8.1数据挖掘标准与规范,DataPreparationPhaseIntegrateDataJoiningmultipledatatablesSummarisation/aggregationofdataSelectDataAttributesubsetselectionRationaleforInclusion/ExclusionDatasamplingTraining/ValidationandTestsets,8.1数据挖掘标准与规范,DataPreparationPhaseDataTransformationUsingfunctionssuchaslogFactor/PrincipalComponentsanalysisNormalization/Discretisation/BinarisationCleanDataHandlingmissingvalues/OutliersDataConstructionDerivedAttributes,8.1数据挖掘标准与规范,TheModelingPhaseBuildModelChooseinitialparametersettingsStudymodelbehaviour:SensitivityanalysisAssessthemodelBewareofover-fittingInvestigatetheerrordistribution:IdentifysegmentsofthestatespacewherethemodelislesseffectiveIterativelyadjustparametersettings,8.1数据挖掘标准与规范,TheEvaluationPhaseValidateModelHumanevaluationofresultsbydomainexpertsEvaluateusefulnessofresultsfrombusinessperspectiveDefinecontrolgroupsCalculateliftcurvesExpectedReturnonInvestmentReviewProcessDeterminenextstepsPotentialfordeploymentDeploymentarchitectureMetricsforsuccessofdeployment,8.1数据挖掘标准与规范,PMML（预测模型标记语言，PredictiveModelMarkupLanguage）。数据挖掘应用往往需要多种类型的数据挖掘软件、算法协同运行，这就要求对挖掘出的模型能够很好地继承、复用与集成。DMG（TheDataMiningGroup，DMG）提出PMML语言。PMML最新版本为4.1，支持16种数据挖掘模型，包括：AssociationModel（关联规则）、BaselineModel（基准模型）、ClusteringModel（聚类模型）、GeneralRegressionModel（回归模型）、MiningModel（组合模型）、NaiveBayesModel（朴素贝叶斯）、NearestNeighborModel（最近邻模型）NeuralNetwork（神经网络）、RegressionModel（线性、多项式、对数三种回归模型）、RuleSetModel（规则集）、SequenceModel（序列模式）、Scorecard、TimeSeriesModel、SupportVectorMachineModel（支持向量机）、TextModel（文本模型）、TreeModel（决策树）,8.1数据挖掘标准与规范,PMML的模型定义由以下几部分组成：,8.1数据挖掘标准与规范,TheheaderelementcontainsgeneralinformationaboutthePMMLdocument,suchascopyrightformationforthemodel,itsdescription,andinformationabouttheapplicationusedtogeneratethemodelsuchasnameandversion.,8.1数据挖掘标准与规范,Thedatadictionaryrecordsinformationaboutthedataeldsfromwhichthemodelwasbuilt.,8.1数据挖掘标准与规范,DataTransformations:transformationsallowforthemappingofuserdataintoamoredesirableformtobeusedbytheminingmodel.PMMLdefinesseveralkindsofsimpledatatransformations.Normalization:mapvaluestonumbers,theinputcanbecontinuousordiscrete.Discretization:mapcontinuousvaluestodiscretevalues.Valuemapping:mapdiscretevaluestodiscretevalues.Functions(customandbuilt-in):deriveavaluebyapplyingafunctiontooneormoreparameters.Aggregation:usedtosummarizeorcollectgroupsofvalues.,8.1数据挖掘标准与规范,Model:containsthedefinitionofthedataminingmodel.ModelName(attributemodelName)AlgorithmName(attributealgorithmName)NumberofLayers(attributenumberOfLayers)MiningSchema:listsallfieldsusedinthemodel.Name:mustrefertoafieldinthedatadictionaryUsagetype:definesthewayafieldistobeusedinthemodel.Typicalvaluesare:active,predicted,andsupplementary.Predictedfieldsarethosewhosevaluesarepredictedbythemodel.OutlierTreatment:definestheoutliertreatmenttobeuse.MissingValueReplacementPolicy:ifthisattributeisspecifiedthenamissingvalueisautomaticallyreplacedbythegivenvalues.MissingValueTreatment:indicateshowthemissingvaluereplacementwasderived.,8.1数据挖掘标准与规范,Targets:allowforpost-processingofthepredictedvalueintheformatofscalingiftheoutputofthemodeliscontinuous.,8.1数据挖掘标准与规范,PMMLExample:AssociationRule：,8.1数据挖掘标准与规范,t1:Cracker,Coke,Watert2:Cracker,Watert3:Cracker,Watert4:Cracker,Coke,Water,Modelattributes,Items,PMMLExample:AssociationRule：,8.1数据挖掘标准与规范,t1:Cracker,Coke,Watert2:Cracker,Watert3:Cracker,Watert4:Cracker,Coke,Water,ItemSets,AssociationRules,JDM（JavaDataMiningAPI）。旨在提供一个访问数据挖掘工具的标准API，支持数据挖掘模型的建立、使用，数据及元数据的创建、存储、访问及维护，从而使得Java应用程序能够能够方便集成数据挖掘技术。,8.1数据挖掘标准与规范,SemanticWeb相关标准TimBerners-Lee在XML2000会议报告中首次提出了语义Web的层次模型（LayerCake）。其特点在与：基于XML和RDF/RDFS，构建本体和逻辑推理规则，以完成基于语义的知识表示和推理，从而为计算机所理解和处理。,8.1数据挖掘标准与规范,第一层是Unicode（统一编码）和URI（UniformResourceIdentifier，统一资源标识器）。UNICODE于1993年成为国际标准组织ISO的一项国际标准ISO/IEC10646，其宗旨是全球所有文种统一编码。URI包含三个部分：被用来访问资源的统一命名规则分配体系、资源宿主机器的名称、路径形式的资源名称。与URL本不同的是，URI只是一个标识符，不直接提供访问资源的方法。,8.1数据挖掘标准与规范,第二层是XML（EXtensibleMarkupLanguage）。XML具有简单、自描述、可扩展的特点，并且实现了内容、结构和表现三者的分离，因而，更适合于数据表示和交换。XMLSchema中的约束主要用于XML文档的结构合法性验证。第三层是RDF（ResourceDescriptionFramework，资源描述框架）。元数据层。RDF是建立在XML上的元数据描述与交换框架，以“资源（Resource）属性（Property）属性值（PropertyValue）”的形式描述对象。一个例子,8.1数据挖掘标准与规范,8.1数据挖掘标准与规范,8.1数据挖掘标准与规范,第四层是RDF-S（RDFSchema）。RDF-S是对RDF的扩展，是RDF的词汇描述语言（VocabularyDescriptionLanguage），用于定义RDF资源描述文件中出现的词汇。第五层是本体（Ontology）和规则（Rule）。领域知识层。OWL用于明确表示词汇体系中的术语及术语间的关系，在词义和语义的表达来说，OWL有更强的表达能力。规则用于描述领域知识中的前提和结论。SPARQL（SimpleProtocolandRDFQueryLanguage）是W3C推荐的用于对RDF数据查询的语言和协议。,8.1数据挖掘标准与规范,本章内容,8.1数据挖掘标准与规范8.2数据挖掘工具8.3数据挖掘的研究趋势,Freeopen-sourcedataminingsoftwareandapplicationsGATE:anaturallanguageprocessingandlanguageengineeringtool.Orange:Acomponent-baseddataminingandmachinelearningsoftwaresuitewritteninthePythonlanguage.R:Aprogramminglanguageandsoftwareenvironmentforstatisticalcomputing,datamining,andgraphics.RapidMiner:Anenvironmentformachinelearninganddataminingexperiments.UIMA:TheUIMA(UnstructuredInformationManagementArchitecture)isacomponentframeworkforanalyzingunstructuredcontentsuchastext,audioandvideooriginallydevelopedbyIBM.Weka:AsuiteofmachinelearningsoftwareapplicationswrittenintheJavaprogramminglanguage.,8.2数据挖掘工具,Commercialdata-miningsoftwareandapplicationsIBMSPSSModeler:dataminingsoftwareprovidedbyIBM.MicrosoftAnalysisServices:dataminingsoftwareprovidedbyMicrosoft.OracleDataMining:dataminingsoftwarebyOracle.SASEnterpriseMiner:dataminingsoftwareprovidedbytheSASInstitute.STATISTICADataMiner:dataminingsoftwareprovidedbyStatSoft.,8.2数据挖掘工具,WEKA:WaikatoEnvironmentforKnowledgeAnalysisItsadatamining/machinelearningtooldevelopedbyDepartmentofComputerScience,UniversityofWaikato,NewZealand.WekaisalsoabirdfoundonlyontheislandsofNewZealand.DownloadandInstallWEKAWebsite:http:/www.cs.waikato.ac.nz/ml/weka/index.htmlSupportmultipleplatforms(writteninjava):Windows,MacOSXandLinux,8.2数据挖掘工具,MainFeatures49datapreprocessingtools76classification/regressionalgorithms8clusteringalgorithms3algorithmsforfindingassociationrules15attribute/subsetevaluators+10searchalgorithmsforfeatureselectionMainGUI“TheExplorer”(exploratorydataanalysis)“TheExperimenter”(experimentalenvironment)“TheKnowledgeFlow”(newprocessmodelinspiredinterface),8.2数据挖掘工具,WEKAonlydealswith“flat”files,8.2数据挖掘工具,relationheart-disease-simplifiedattributeagenumericattributesexfemale,maleattributechest_pain_typetyp