商务智能应用-统计分析的进阶加值应用

由知识挖掘提升商务智能应用-统计分析的进阶加值应用FromKnowledgeMiningtoBusinessIntelligence-AdvancedStatisticsApplication,谢邦昌博士厦门大学讲座教授兼博导首都经贸大学讲座教授兼博导中央财经大学讲座教授兼博导西南财经大学讲座教授中国人民大学兼职教授辅仁大学统计资讯学系及应用统计所教授中华资料采矿协会理事长,Outline,知识采矿(整合数据采矿与文本采矿)与商业智慧的发展知识采矿程序、步骤、产出与应用如何进行数据采矿与文本采矿整合知识采矿之技术发展评论,知识保存价值,减少循环时间反应时间重复投资作业花费会议时间外界顾问等等,增加生产力与质量企业知识的转换快且有效的决策课程创新群策群力等等,企业知识的保留与转换知识资产的投资精简与退休人员轮替生产力能力重复能量消耗过多的会议沟通问题组织目标下达决策可行性快速非正规,为何知识如此迫切?,“Thechiefeconomicpriorityfordevelopedcountriesistoraisetheproductivityofknowledge.Thecountrythatdoesthisfirstwilldominatethetwenty-firstcenturyeconomically.”开发中国家首要经济目标为知识的创造力谁先掌握谁就统领二十一世纪的经济,PeterF.Drucker,资料知识形成流程,Integration,RawData,Understanding,BI结构,Monitorastart-upventurefullycapitalizedbyaGlobalLeaderinAdvancedTechnologies.Qualifiedcandidateswill:Responsibleforassistingwithrequirementsdefinition,analysis,designandimplementationthatmeetobjectives,codesdifficultandsophisticatedroutines.Developsprojectplans,schedulesandcostdata.Developtestplansandimplementphysicaldesignofdatabases.Developshellscriptsforadministrativeandbackgroundtasks,storedproceduresandtriggers.UsingOraclesDesigner2000,assistwithDataModelmaintenanceandassistwithapplicationsdevelopmentusingOracleForms.Qualifications:BSCS,BSMISorcloselyrelatedfieldorrelatedequivalentknowledgenormallyobtainedthroughtechnicaleducationprograms.5-8yearsofprofessionalexperienceindevelopment,systemdesignanalysis,programming,installationusingOracledevelopment,AutomaticPattern-LearningSystems,Pros:PortableacrossdomainsTendtohavebroadcoverageRobustinthefaceofdegradedinput.AutomaticallyfindappropriatestatisticalpatternsSystemknowledgenotneededbythosewhosupplythedomainknowledge.Cons:Annotatedtrainingdata,andlotsofit,isneeded.Isntnecessarilybetterorcheaperthanhand-builtsolnExamples:Riloffetal.,AutoSlog,SoderlandWHISK(UMass);Mooneyetal.Rapier(UTexas);Ciravegna(Sheffield)Learnlexicon-syntacticpatternsfromtemplates,TextAnalysisSpectrum,EntityExtraction,TargetedFactsandEvents,Classification,Clustering,ConceptIdentification,Whatisthisdocumentabout?,Whodidwhattowhomwhenwhere,etc.,Whyisgettingdimensionaldatasohard?,HankboughtplasticexplosivesfromHenryinTucsonyesterday.,NamedEntityExtraction,People,Weapons,Vehicles,Dates,NEREngine,Hank,Henry,Plasticexplosives,Tucson,11/01/07,FrameNet,NameExtractionviaMMs,Text,SpeechRecognition,Extractor,Speech,Entities,NEModels,Thedelegation,whichincludedthecommanderoftheU.N.troopsinBosnia,Lt.Gen.SirMichaelRose,wenttotheSerbstrongholdofPale,nearSarajevo,fortalkswithBosnianSerbleaderRadovanKaradzic.,TrainingProgram,trainingsentences,answers,Thedelegation,whichincludedthecommanderoftheU.N.troopsinBosnia,Lt.Gen.SirMichaelRose,wenttotheSerbstrongholdofPale,nearSarajevo,fortalkswithBosnianSerbleaderRadovanKaradzic.,AneasybutsuccessfulHMMapplication:Priorto1997-nolearningapproachcompetitivewithhand-builtrulesystemsSince1997-Statisticalapproaches(BBN(Bikeletal.1997),NYU,MITRE,CMU/JustSystems)achievestate-of-the-artperformance,NER,数据库探勘作业流程,概念分群,Anopheles,FeedbackasModelInterpolation,非单调性资料(Heterogeneous),文件分群,Mooter,科学人杂志3月号,文件数据分群,AnnotationandTagging,OnNovember16,2005,IBMannouncedithadacquiredCollation,aprivatelyheldcompanybasedinRedwoodCity,Californiaforundisclosedamount.,Date,AcquiringOrganization,AcquisitionEvent,AcquiredOrganization,Place,Amount,TextAnnotator,OutputtoRDBMS,XMLoutput,OnNovember16,2005,IBMannouncedithadacquiredCollation,aprivatelyheldcompanybasedinRedwoodCity,Californiaforundisclosedamount.,LinguisticConceptExtractionfromCustomerServiceRecords,Bagof“Words”extraction,CstmrIDCustomerYellowIncHappyNotSwitchCellPhone,Expressionsextraction,CstmrIDCustomerYellowIncswitchCellPhoneNothappy,NamedEntitiesextraction,CustomerCRMtermCstmr?YellowIncTelcoCompanyCellPhoneTelcoTermNothappySwitch,Events/SentimentExtraction,Customer(cstmr)cellphoneunhappy(Negative)Switchto(NegativePredicate)yellowinc(Competition),CombinedWithstructureddata,DecisionMakingChurnerSpecialOffer,KnowledgeInference,InformationExtraction,InformationRetrieval,ExtractingInformationFromText,Structuringknowledgefromtexttagging,compounds,grammaticalanalysis,ontologicalinterpretation,regularexpressions,patterrecognition,Database,Minimalrecursionsemanticsrepresentations,DeepThoughtEUproject,KnowledgeConstruction,Wanttoextractprominentconcepts/relationsfromtexttagging,compounds,NPrecognition,termfrequencies,stopwords,languageidentification,Brasethvik&Gulla,DKE,38/1,2001,PatternsConstruction,Taipei,Tokyo,NewYork,Repository,Tagging&annotation,CDW,KnowledgeRepositoryOrstructureddata,Patterns,Patterns,ExplorerWebBrowser,Harddisk,WindowsXP,DesktopcomputerHarddisksize40GB,Products,Laptopcomputers,OperatingSystem,Linux,Macintosh,isa,crashes,Installedfromhttp:/.,人、事、时、地、物元资料,人物,性质,TemporalEntities,应用,referto/identifie,at,within,资源索引,人物,事件,物件,Derivedknowledgedata(e.g.RDF),ThesauriextentCRMentities,Ontologyexpansion,Sourcesandmetadata(XML/RDF),Backgroundknowledge/Authorities,CIDOCCRMorDC,ConceptLattice,C1:(D1,),C2:(d1,d2,d4,t1,t6),C3:(d3,d4,t4),C4:(d1,d2,t1,t3,t5,t6),C5:(d4,t1,t4,t6),C6:(d3,t2,t4),C7:(,T1),TheformalconceptC4hastwoowntermst3,t5andtwoinheritedtermst1,t6,Giventhecontext(D1,T1)whereD1=d1,d2,d3,d4&T1=t1,t2,t3,t4,t5,t6,Rt1t2t3t4t5t6d1101011d2101011d3010100d4100101,Table:TheinputrelationR=documentskeywords,HasseDiagram,P14performed,P11participatedin,P94hascreated,E7Activity,“CrimeaConference”,E65CreationEvent,*,P86fallswithin,P7tookplaceat,P67isreferredtoby,P81ongoingthroughout,P82atsometimewithin,ExplicitEvents,ObjectIdentity,Symmetry,RulesExtraction,TheformalconceptC4makesitpossiblethefollowingrulesR1:t3t1t6R2:t5t1t6R3:t3t5TheinterpretationoftheR1andR2:Theuseoftermst3ort5isalwaysassociatedwiththatoftermst1andt6TheruleR3expressmutualequivalenceofthetermst3,t5:Allthedocumentswhichhavethetermt3alsohavethet5term.,知识群组,专家与决策,知识呈现,实时性分群,Real-timeIndex,MetadataofSearchingResults,公文性资料,因果图-失依儿童,各县市福利，信托基金的成立,所在各县市失依儿童状态,各县市政府，社会局等介入,对单亲家庭的补助之灾后重建及经费相关使用,灾后重建基金,规则,Clustering,范例,知识脉络,知识地图,事件追踪,信息检索,知识概念,KuhnsDescriptiveProject,ImmatureScience,NormalScience,Anomalies,Crisis,Revolution,Evolutionarytheoryisevolving,TasksinNewsDetection,NewsFeeds,Detection,Segmentation,On-Line,Retro,Tracking,MightbeRelevant,USSColeOctober12,2000,世贸中心五角大厦2001年九月11日,911事件,可预防FBI明尼苏达干员ZacariasMoussaoui个人计算机FBI凤凰城备忘录(GeorgeWill)Dr.Bhandari(VirtualGold,Inc)资料探勘可预防911悲剧,恐怖份子,911恐怖份子网络,911恐怖份子网络,赤军旅(RedArmyFaction)威胁,HorstHerold(德国联邦警察总长)建立数据探勘之信息网GermanysBundeskriminalamt1972数据源房屋销售、能源公司成果RolfHeissler(RAF成员)结果erold遭报导违反人权退休1986修改犯罪条例911三个飞行员系来自Hamburg,疫病警示及通报系统,世界卫生组织多年前即建立了疫病警示及通报系统（EpidemicAlertandResponse）。由于一些国家可能基于经济冲击的考虑，可能淡化有关疫情的报导，世界卫生组织的这套系统特别装置了一套软件，可以由各国媒体的网站上抓取相关资料并由二十位专家分析这些资料中的信息。,HighW,信息与知识Amazon数字相机销售,新闻事件华盛顿时报,美国家卫生院NIH热门研究,ProposalsbyFunding/DateacrossIRGsandActivityTypes,疾病诊疗指引Athena/EON-Stanford,Athena临床指引,R.D.Shankar,etal.2001,高血压临床指引AthenaHypertensionGuideline,A.Advani,etal.2003,受灾户(金融辅助政策),贷款(受灾户、临时住宅),GenerativeDiscriminative,重建家园专案,金融机构,贷款,震灾重建暂行条例,受灾户,房屋,利息,损毁,灾户,object,method,Objec