8-1数据仓库与数据挖掘

1、Chapter 20: Data Analysis Chapter 20:Data AnalysisDecisionSupport SystemsData WarehousingData MiningClassificationAssociationRulesClusteringDecisionSupport SystemsDecision-support systemsareusedtomake business decisions,oftenbasedondatacollectedbyon-linetransaction-processing systems.Examplesofbusin

2、essdecisions:What items to stock?What insurancepremium to change?Towhom to sendadvertisements?Examplesofdata usedfor makingdecisionsRetailsalestransactiondetailsCustomerprofiles(income,age, gender,etc.)Decision-Support Systems: OverviewData analysistasksaresimplifiedbyspecializedtoolsandSQL extensio

3、nsExample tasksForeachproduct category andeach region,whatwere thetotalsalesinthelastquarter andhowdotheycompare withthe samequarterlast yearAsabove,for eachproductcategoryandeachcustomercategoryStatisticalanalysispackages(e.g.,:S+) canbeinterfaced withdatabasesStatisticalanalysisisa large field,but

4、not coveredhereData miningseekstodiscoverknowledgeautomaticallyintheformofstatisticalrulesandpatternsfromlargedatabases.Adata warehousearchivesinformationgatheredfrom multiple sources, andstoresitunderaunified schema,atasinglesite.Importantfor large businessesthatgeneratedata frommultipledivisions,p

5、ossiblyatmultiplesitesData mayalso be purchasedexternallyData WarehousingData sourcesoftenstoreonlycurrent data, nothistorical dataCorporatedecisionmaking requires aunifiedview of allorganizationaldata,includinghistoricaldataAdata warehouseisa repository(archive) of information gathered frommultiple

6、sources,stored under aunifiedschema, at asingle siteGreatly simplifiesquerying, permitsstudyofhistoricaltrendsShiftsdecisionsupportqueryload awayfromtransactionprocessing systemsData WarehousingDesignIssuesWhen andhowtogather dataSourcedriven architecture: datasourcestransmitnewinformationtowarehous

7、e,eithercontinuously or periodically(e.g.,atnight)Destinationdrivenarchitecture: warehouseperiodicallyrequestsnew information fromdatasourcesKeeping warehouseexactly synchronizedwith datasources(e.g.,usingtwo-phasecommit)istooexpensiveUsually OK to haveslightlyout-of-datedataatwarehouseData/updatesa

8、re periodicallydownloaded formonline transaction processing(OLTP) systems.What schematouseSchemaintegrationMore WarehouseDesignIssuesData cleansingE.g.,correct mistakes in addresses(misspellings,zipcodeerrors)Mergeaddress lists fromdifferent sourcesandpurgeduplicatesHowtopropagate updatesWarehousesc

9、hema maybea (materialized) viewofschema fromdatasourcesWhat datatosummarizeRawdatamaybetoo large to store on-lineAggregatevalues (totals/subtotals)oftensufficeQueries on rawdata canoftenbetransformedbyqueryoptimizertouse aggregatevaluesWarehouseSchemasDimensionvalues areusually encodedusingsmallinte

10、gersand mappedtofull valuesviadimension tablesResultantschema is calledastar schemaMore complicated schemastructuresSnowflakeschema: multiple levelsofdimensiontablesConstellation: multiple facttablesData WarehouseSchemaData MiningData miningistheprocessofsemi-automaticallyanalyzing large databasesto

11、find usefulpatternsPredictionbasedonpast historyPredict if acredit cardapplicant poses agoodcreditrisk,basedonsomeattributes (income, jobtype,age, .)andpasthistoryPredict if apatternofphonecalling cardusageislikely to be fraudulentSome examples of predictionmechanisms:ClassificationGivena newitem wh

12、ose class is unknown, predicttowhichclassitbelongsRegressionformulaeGivena setofmappingsforanunknownfunction,predictthefunctionresult fora newparametervalueData Mining(Cont.)DescriptivePatternsAssociationsFind books thatare often boughtby“similar”customers.Ifanewsuchcustomerbuys onesuch book, sugges

13、tthe otherstoo.Associationsmay be usedasafirststep in detectingcausationE.g.,associationbetween exposure to chemical Xand cancer,ClustersE.g.,typhoid cases wereclustered in an areasurroundingacontaminatedwellDetectionofclustersremainsimportantindetecting epidemicsClassificationRulesClassificationrul

14、eshelp assignnewobjectstoclasses.E.g.,givena newautomobile insuranceapplicant, shouldheorshebeclassifiedaslowrisk,medium riskorhighrisk?Classificationrulesforaboveexamplecoulduseavariety of data, suchaseducationallevel, salary,age,etc.personP,P.degree =mastersandP.income 75,000P.credit= excellentper

15、sonP,P.degree =bachelorsand(P.income25,000and P.income75,000)P.credit= goodRulesarenot necessarily exact:theremaybesomemisclassificationsClassificationrulescanbeshowncompactly as adecisiontree.DecisionTreeConstructionofDecisionTreesTrainingset: adatasampleinwhichthe classification is alreadyknown.Gr

16、eedytopdowngeneration of decision trees.Each internal nodeofthe treepartitionsthedatainto groupsbasedonapartitioningattribute, andapartitioningconditionforthe nodeLeafnode:all(or most) of theitemsatthenodebelongtothe sameclass, orallattributeshave beenconsidered,and no furtherpartitioning is possibl

17、e.Best SplitsPick bestattributesandconditionsonwhichtopartitionThepurity of aset Softraininginstances canbemeasuredquantitativelyinseveral ways.Notation:number of classes=k,numberofinstances =|S|,fractionofinstances in classi=pi.TheGinimeasure of purityisdefined asGini (S)= 1-When allinstancesare in

18、 asingle class,theGinivalueis0Itreaches itsmaximum (of1 1 /k) if eachclassthe samenumber of instances. ki- 1p2iBest Splits(Cont.)Another measureofpurity is theentropymeasure,whichisdefined asentropy (S)= When aset Sissplitintomultiplesets Si,I=1, 2, , r, we canmeasure thepurityofthe resultantsetofse

19、tsas:purity(S1, S2, .,Sr) =TheinformationgainduetoparticularsplitofS intoSi, i=1,2,.,rInformation-gain(S, S1,S2, .,Sr) =purity(S) purity (S1,S2, Sr)ri= 1|Si|S|purity (Si)ki- 1pilog2 piBest Splits(Cont.)Measure of “cost”ofa split:Information-content(S, S1,S2, .,Sr)=Information-gain ratio=Information-

20、gain (S,S1,S2, ,Sr)Information-content (S, S1,S2, .,Sr)Thebestsplitistheone thatgivesthe maximuminformationgainratiolog2ri- 1|Si|S|Si|S| Finding BestSplitsCategoricalattributes (with no meaningfulorder):Multi-waysplit, onechildforeachvalueBinarysplit: tryallpossiblebreakupofvaluesintotwosets,and pic

21、kthe bestContinuous-valuedattributes (canbesorted in ameaningfulorder)Binarysplit:Sort values,try eachasasplitpointE.g.,ifvaluesare 1, 10,15,25, split at1, 10, 15Pick thevaluethat gives bestsplitMulti-waysplit:A seriesofbinarysplits on thesame attributehasroughlyequivalent effectDecision-Tree Constr

22、uctionAlgorithmProcedureGrowTree(S)Partition(S);ProcedurePartition(S)if(purity(S) por|S| s)thenreturn;foreachattributeAevaluatesplitsonattributeA;Usebestsplitfound(acrossallattributes)topartitionSintoS1, S2, .,Sr,fori= 1, 2, .,rPartition(Si);OtherTypesofClassifiersNeuralnet classifiers arestudied in

23、 artificialintelligence andarenot coveredhereBayesianclassifiersuseBayestheorem, which saysp(cj|d) =p(d| cj)p(cj)p(d)wherep(cj|d) =probabilityofinstancedbeinginclasscj,p(d| cj) =probabilityofgeneratinginstancedgivenclasscj,p(cj)= probability of occurrenceofclasscj, andp(d) =probabilityofinstancedocc

24、uringNaveBayesianClassifiersBayesianclassifiersrequirecomputationofp(d| cj)precomputationofp(cj)p(d) canbeignored since it is thesame forallclassesTosimplifythetask,naveBayesianclassifiersassumeattributeshave independent distributions, andthereby estimatep(d|cj) =p(d1|cj) *p(d2|cj) *.* (p(dn|cj)Each

25、 of thep(di|cj) canbeestimatedfroma histogramondivaluesfor eachclasscjthehistogram is computed fromthe training instancesHistograms on multiple attributesare moreexpensive to computeand storeRegressionRegression deals withthe predictionofavalue,rather thanaclass.Givenvaluesfor aset of variables,X1,

26、X2, , Xn, we wishtopredictthevalueofavariableY.Oneway is to infer coefficientsa0, a1, a1, , ansuch thatY=a0+a1*X1+a2*X2+ +an*XnFinding suchalinearpolynomialiscalledlinearregression.Ingeneral,theprocessoffinding acurvethatfits thedata is alsocalledcurvefitting.Thefit mayonly be approximatebecause of

27、noise in thedata,orbecause therelationshipisnot exactlyapolynomialRegression aimstofindcoefficientsthatgive thebest possible fit.AssociationRulesRetailshopsare often interestedinassociations betweendifferent items thatpeople buy.Someone whobuys bread is quite likelyalso to buymilkA personwhobought t

28、hebookDatabaseSystemConceptsisquitelikelyalsotobuythe bookOperatingSystem Concepts.Associationsinformationcan be usedinseveralways.E.g.,when acustomerbuysa particularbook,anonline shopmay suggestassociatedbooks.Associationrules:breadmilkDB-Concepts,OS-Concepts NetworksLeft handside:antecedent,righth

29、and side:consequentAnassociationrule musthaveanassociatedpopulation; thepopulation consists of aset ofinstancesE.g.,each transaction (sale)ata shopisaninstance, andtheset of alltransactionsisthe populationAssociationRules(Cont.)Ruleshave an associatedsupport,aswellasanassociated confidence.Supportis

30、a measureofwhatfractionofthepopulationsatisfiesboththeantecedentandthe consequentofthe rule.E.g.,suppose only0.001percentofallpurchases includemilkandscrewdrivers.Thesupportforthe ruleismilkscrewdriversislow.Confidenceisa measureofhow often theconsequent is truewhentheantecedentistrue.E.g.,therulebr

31、eadmilkhasaconfidence of 80 percentif80percentofthepurchases thatincludebreadalso includemilk.Finding Association RulesWearegenerally onlyinterestedinassociationruleswith reasonablyhighsupport (e.g.,support of 2% or greater)NavealgorithmConsiderall possible setsofrelevantitems.Foreachsetfinditssuppo

32、rt(i.e.,counthow manytransactionspurchaseallitemsinthe set).Largeitemsets: setswithsufficientlyhighsupportUselargeitemsetstogenerateassociationrules.From itemsetAgeneratetheruleA- b bforeachbA.Support of rule=support (A).Confidence of rule=support (A) /support(A- b)Finding SupportDeterminesupportofi

33、temsetsviaasinglepassonsetoftransactionsLargeitemsets:setswith ahighcountattheend of thepassIfmemorynot enoughtohold allcountsfor allitemsetsusemultiplepasses,consideringonly someitemsetsineachpass.Optimization: Onceanitemsetiseliminated becauseits count (support)istoosmallnoneofitssupersets needs t

34、o be considered.Thea prioritechniquetofindlargeitemsets:Pass 1: count supportofall setswithjust 1item.Eliminatethoseitemswith lowsupportPassi:candidates: every setofiitemssuch thatall itsi-1item subsetsare largeCountsupport of allcandidatesStop if there arenocandidatesOtherTypesofAssociationsBasicas

35、sociationruleshave severallimitationsDeviations fromthe expected probability aremore interestingE.g.,ifmany peoplepurchasebread,and manypeople purchase cereal,quiteafewwouldbeexpectedtopurchasebothWeareinterestedinpositiveaswell asnegativecorrelationsbetween setsofitemsPositivecorrelation:co-occurre

36、nceishigherthanpredictedNegativecorrelation:co-occurrenceislowerthan predictedSequenceassociations/correlationsE.g.,wheneverbondsgoup,stockprices go downin2daysDeviations fromtemporalpatternsE.g.,deviationfroma steadygrowthE.g.,salesofwinterweargodown in summerNotsurprising,partofa known pattern.Loo

37、k fordeviationfromvaluepredictedusingpastpatternsClusteringClustering:Intuitively,findingclustersofpointsinthe given datasuchthat similarpoints lieinthesameclusterCanbeformalizedusingdistancemetrics in severalwaysGrouppointsintoksets (foragivenk) suchthattheaveragedistanceofpointsfromthecentroidofth

38、eirassignedgroupisminimizedCentroid:pointdefinedbytakingaverageofcoordinatesineach dimension.Another metric:minimizeaveragedistancebetween every pairofpoints in aclusterHasbeenstudied extensively in statistics, butonsmalldata setsData miningsystems aimatclustering techniquesthatcanhandle verylargeda

39、tasetsE.g.,theBirchclusteringalgorithm(moreshortly)HierarchicalClusteringExample frombiologicalclassification(the wordclassificationheredoes notmean apredictionmechanism)chordatamammaliareptilialeopardshumanssnakescrocodilesOtherexamples:Internetdirectory systems(e.g., Yahoo,more on thislater)Agglom

40、erative clusteringalgorithmsBuildsmallclusters,thencluster small clusters intobigger clusters,andsoonDivisiveclustering algorithmsStartwith allitemsina singlecluster,repeatedly refine(break) clusters intosmalleronesClustering AlgorithmsClustering algorithmshavebeen designed to handlevery large datasetsE.g.,theBirchalgorithmMain idea: useanin-memoryR-tree to store pointsthat arebeingclusteredInsertpoints oneata timeintotheR