大数据数据挖掘培训讲义3：概念,属性和实例

Input:Concepts,Attributes,Instances,2,ModuleOutline,TerminologyWhatsaconcept?Classification,association,clustering,numericpredictionWhatsinanexample?Relations,flatfiles,recursionWhatsinanattribute?Nominal,ordinal,interval,ratioPreparingtheinputARFF,attributes,missingvalues,gettingtoknowdata,witten&eibe,3,Terminology,Componentsoftheinput:Concepts:kindsofthingsthatcanbelearnedAim:intelligibleandoperationalconceptdescriptionInstances:theindividual,independentexamplesofaconceptNote:morecomplicatedformsofinputarepossibleAttributes:measuringaspectsofaninstanceWewillfocusonnominalandnumericones,witten&eibe,4,Whatsaconcept?,DataMiningTasks(Stylesoflearning):Classificationlearning:predictingadiscreteclassAssociationlearning:detectingassociationsbetweenfeaturesClustering:groupingsimilarinstancesintoclustersNumericprediction:predictinganumericquantityConcept:thingtobelearnedConceptdescription:outputoflearningscheme,witten&eibe,5,Classificationlearning,Exampleproblems:attritionprediction,usingDNAdatafordiagnosis,weatherdatatopredictplay/notplayClassificationlearningissupervisedSchemeisbeingprovidedwithactualoutcomeOutcomeiscalledtheclassoftheexampleSuccesscanbemeasuredonfreshdataforwhichclasslabelsareknown(testdata)Inpracticesuccessisoftenmeasuredsubjectively,6,Associationlearning,Examples:supermarketbasketanalysis-whatitemsareboughttogether(k+cereal,chips+salsa)Canbeappliedifnoclassisspecifiedandanykindofstructureisconsidered“interesting”Differencewithclassificationlearning:Canpredictanyattributesvalue,notjusttheclass,andmorethanoneattributesvalueatatimeHence:farmoreassociationrulesthanclassificationrulesThus:constraintsarenecessaryMinimumcoverageandminimumaccuracy,7,Clustering,Examples:customergroupingFindinggroupsofitemsthataresimilarClusteringisunsupervisedTheclassofanexampleisnotknownSuccessoftenmeasuredsubjectively,witten&eibe,8,Numericprediction,Classificationlearning,but“class”isnumericLearningissupervisedSchemeisbeingprovidedwithtargetvalueMeasuresuccessontestdata,witten&eibe,9,Whatsinanexample?,Instance:specifictypeofexampleThingtobeclassified,associated,orclusteredIndividual,independentexampleoftargetconceptCharacterizedbyapredeterminedsetofattributesInputtolearningscheme:setofinstances/datasetRepresentedasasinglerelation/flatfileRatherrestrictedformofinputNorelationshipsbetweenobjectsMostcommonforminpracticaldatamining,witten&eibe,10,Afamilytree,PeterM,PeggyF,=,StevenM,GrahamM,PamF,GraceF,RayM,=,IanM,PippaF,BrianM,=,AnnaF,NikkiF,witten&eibe,11,Familytreerepresentedasatable,witten&eibe,12,The“sister-of”relation,Closed-worldassumption,witten&eibe,13,Afullrepresentationinonetable,witten&eibe,14,Generatingaflatfile,Processofflatteningafileiscalled“denormalization”SeveralrelationsarejoinedtogethertomakeonePossiblewithanyfinitesetoffiniterelationsProblematic:relationshipswithoutpre-specifiednumberofobjectsExample:conceptofnuclear-familyDenormalizationmayproducespuriousregularitiesthatreflectstructureofdatabaseExample:“supplier”predicts“supplieraddress”,witten&eibe,15,*The“ancestor-of”relation,witten&eibe,16,*Recursion,Appropriatetechniquesareknownas“inductivelogicprogramming”(e.g.QuinlansFOIL)Problems:(a)noiseand(b)computationalcomplexity,Infiniterelationsrequirerecursion,witten&eibe,17,*Multi-instanceproblems,EachexampleconsistsofseveralinstancesE.g.predictingdrugactivityExamplesaremoleculesthatareactive/notactiveInstancesareconfirmationsofamoleculeMoleculeactive(examplepositive)catleastoneofitsconfirmations(instances)isactive(positive)Moleculenotactive(examplenegative)callofitsconfirmations(instances)arenotactive(negative)Problem:identifyingthe“truly”positiveinstances,witten&eibe,18,Whatsinanattribute?,Eachinstanceisdescribedbyafixedpredefinedsetoffeatures,its“attributes”But:numberofattributesmayvaryinpracticePossiblesolution:“irrelevantvalue”flagRelatedproblem:existenceofanattributemaydependofvalueofanotheronePossibleattributetypes(“levelsofmeasurement”):Nominal,ordinal,intervalandratio,witten&eibe,19,Nominalquantities,ValuesaredistinctsymbolsValuesthemselvesserveonlyaslabelsornamesNominalcomesfromtheLatinwordfornameExample:attribute“outlook”fromweatherdataValues:“sunny”,”overcast”,and“rainy”Norelationisimpliedamongnominalvalues(noorderingordistancemeasure)Onlyequalitytestscanbeperformed,witten&eibe,20,Ordinalquantities,ImposeorderonvaluesBut:nodistancebetweenvaluesdefinedExample:attribute“temperature”inweatherdataValues:“hot”“mild”“cool”Note:additionandsubtractiondontmakesenseExamplerule:temperature“sunny”doesnotmakesense,whileTemperature“cool”orHumidity70doesAdditionalusesofattributetype:checkforvalidvalues,dealwithmissing,etc.,26,Transformingordinaltoboolean,Simpletransformationallowsordinalattributewithnvaluestobecodedusingn1booleanattributesExample:attribute“temperature”Betterthancodingitasanominalattribute,Originaldata,Transformeddata,c,witten&eibe,27,Metadata,InformationaboutthedatathatencodesbackgroundknowledgeCanbeusedtorestrictsearchspaceExamples:Dimensionalconsiderations(i.e.expressionsmustbedimensionallycorrect)Circularorderings(e.g.degreesincompass)Partialorderings(e.g.generalization/specializationrelations),witten&eibe,28,Preparingtheinput,Problem:differentdatasources(e.g.salesdepartment,customerbillingdepartment,)Differences:stylesofrecordkeeping,conventions,timeperiods,dataaggregation,primarykeys,errorsDatamustbeassembled,integrated,cleanedup“Datawarehouse”:consistentpointofaccessDenormalizationisnottheonlyissueExternaldatamayberequired(“overlaydata”)Critical:typeandlevelofdataaggregation,witten&eibe,29,TheARFFformat,witten&eibe,30,AttributetypesinWeka,ARFFsupportsnumericandnominalattributesInterpretationdependsonlearningschemeNumericattributesareinterpretedasordinalscalesifless-thanandgreater-thanareusedratioscalesifdistancecalculationsareperformed(normalization/standardizationmayberequired)Instance-basedschemesdefinedistancebetweennominalvalues(0ifvaluesareequal,1otherwise)Integers:nominal,ordinal,orratioscale?,witten&eibe,31,Nominalvs.ordinal,Attribute“age”nominalAttribute“age”ordinal(e.g.“young”“pre-presbyopic”“presbyopic”),witten&eibe,32,Missingvalues,Frequentlyindicatedbyout-of-rangeentriesTypes:unknown,unrecorded,irrelevantReasons:malfunctioningequipmentchangesinexperimentaldesigncollationofdifferentdatasetsmeasurementnotpossibleMissingvaluemayhavesignificanceinitself(e.g.missingtestinamedicalexamination)Mostschemesassumethatisnotthecasec“missing”mayneedtobecodedasadditionalvalue,witten&eibe,33,Missingvalues-example,ValuemaybemissingbecauseitisunrecordedorbecauseitisinapplicableInmedicaldata,valueforPregnant?attributeforJaneismissing,whileforJoeorAnnashouldbeconsideredNotapplicableSomeprogramscaninfermissingvalues,HospitalCheck-inDatabase,34,Inaccuratevalues,Reason:datahasnotbeencollectedforminingitResult:errorsandomissionsthatdontaffectoriginalpurposeofdata(e.g.ageofcustomer)TypographicalerrorsinnominalattributesvaluesneedtobecheckedforconsistencyTypographicalandmeasurementerrorsinnumericattributesoutliersneedtobeidentifiedErrorsmaybedeliberate(e.g.wrongzipcodes)Otherproblems:duplicates,staledata,witten&eibe,35,Precision“Illusion”,Example:geneexpressionmaybereportedasX83=193.3742,butmeasurementerrormaybe+/-20.Actualvalueisin173,213range,soitisappropriatetoroundthedatato190.Dontassumethateveryreporteddigitissignific