计算机专业英语论文

######专业英语结课论文学号：**********姓名：**********论文题目：TheRelationshipandDistinctionBetweenBigDataandDataMining任课教师：************专业名称：计算机技术所属学院：计算机科学与工程学院桂林电子科技大学研究生院**年*月*日TheRelationshipandDistinctionBetweenBigDataandDataMiningStudentID:*Name:*Adviser:*GuilinUniversityofElectronicTechnology**,*Abstract:Inthispaper,dataminingisdiscussedinthecontextofbigdata.Firstly,weelaboratethefactthatbigdataplaysaprimaryroleinattractingacademiccommunity,businessindustryandgovernments.Secondly,theadverseofbigdataisdiscussed,suchasmuchgarbage,heavypollutionanditsdifficultiesinutilization.Finally,wedissectthevalueinbigdata,expoundthetechniquestodiscoverknowledgefrombigdata,andinvestigatethetransformationfromknowledgeintodataintelligences.Keywords:bigdata;datamining;dataintelligenceIntroductionAsdatavolumescontinuetoincreaseexponentially,thedatatsunamicaneasilyoverwhelmtraditionalanalyticstoolsorplatformsdesignedtoingest,analyzeandreport.

Everyday,2.5quintillionbytesofdataarecreatedand90percentofthedataintheworldtodaywereproducedwithinthepasttwoyears[1].Thechallengewearefacingisnotonlyhowtostoreandmanagediversedatabutalsotoeffectivelyanalyzethedatatogaininsightknowledgetomakesmarterdecisions.

Currently,anumberofworkshavebeenpresented.

Theseresearchesintroducebigdata,miningandanalyzingfromdifferentaspects,suchasstatusquo,ideasorimplementations.

Forexample:introducesthe“LambdaArchitecture”whichprovidesageneralpurposeapproachtoimplementarbitraryfunctionsonmassivedatasetinrealtime;ascalabledeepanalyticsplatformhasbeenimplemented.Becauseofthecomplexity,thereisnosingletoolorone-size-fits-allsolutionfordeeplyminingandanalyzingthebigdata.Moreover,extractingvaluableknowledgefrommassivedatasetsrequiresfurtherstudies,experimentsaswellasscalableandsmartservices,programmingtoolsandapplicationsachieved.Theremainderofthispaperisstructuredasfollows.

Section2

elaboratethefactthatbigdataplaysaprimaryroleineveryfields.Thentheadverseofbigdataisdiscussedinsection3.Afteranalyzingthevalueofbigdata,weintroduces

therelated

knowledgeanddevelopmentofdataminingin

section5.In

Section6,theeffectivenessofdataminingisintroduced.Finally,theconclusionfollow.AboutbigdataBigdataiscomplexdatasetthathasthefollowingmaincharacteristics:Volume,Variety,VelocityandVeracity[2][3].

Thesemakeitdifficulttousetheexistingtoolstomanageandmanipulate.Inthesedata,bigdataspecificallyaccountsforthevastmajority.

Bigdataisthebasisofdataandsourceofwisdomforpeopletounderstandthereal-worldthroughtheinformationworld.

BigDataiscloselyrelatedtoapplications[4][5],andbigdataminingisitsprincipalapplication.2.1Fromunderstandingthereal-worldtocreatingtheinformationworldHumancivilizationisaprocessfromunderstandingthereal-worldtocreatingtheinformationworld,whichhasgonethroughthefollowingstages:preliminarysensingtheworld,helpingmemorybyinformation,recordedandinheritedbyinformation,exchangeandcommunicationbyinformationandunderstandingtheworldonceagainbyinformation.Initially,Humantakeadvantageofstonesandshellstocountaccordingtotheprincipleofone-to-one.AndtheytieknotsNotetohelpmemory.Later,Humanusesimplegraphics,drawnotes,andinheritmoreaccuratememorythroughtheirownemotionalprompted.Whenthegraphicsbecomebodyrelativelyfixedcommonsymbol,andassociatewiththewordsinthelanguage,itproducestexts.Textsabstractandgeneralizetheworld,promoteculturalunderstanding,andpreparethenecessaryfoundationforthedevelopmentofscience.Aimedatbreakingthroughtherestrictionswhichthewrittensymbolsdependonartificialcopyingorengraving,Humanusemachinesafterindustrialrevolutiontovolumemechanizedproduction,whichimprovestheefficiencyoftheculturaltransmission.Computercentershigh-speedcomputing,andspinsoffthesoftwarefromthehardware,contributingtothedisseminationofinformation“electronically”and“automatically”.Internetcentersnetwork,interrelatescomputers,breakinglocalinformationrestriction.Mobilecommunicationcentersusers,makingthemachinefollowsuser'smovementsandunboundshumanfromthemachine.InternetofThingscentersapplications,automaticallyidentifiesobjects,toenabletheinformationsharingbetweenthehumanandthings.Cloudcomputingcentersservicebyconsolidatingexpertiseandoptimizingtheallocationofresources.

Bigdatacentersdata,andminesknowledgeintheentiredata,breakingthesamplingrandomnessofthesample[6][7],anddemonstratingonbigdatacenterandmobileterminal.

Theseinformationtechnologiesservefortheunderstandingandtransformingoftherealworld.2.2BigdataisattractingmuchattentionAshumansexploretherealworldthroughscientificresearch,humansunravelthemysteriesintheinformationworldthroughbigdataanddatamining,whichareattractingmuchattentionfromacademia.InMay2011,McKinseypublished“Bigdata:thenextfrontierforinnovation,competition,andproductivity”,analyzedapplicationpotentialofbigdataindifferentindustriesfromtheeconomicandcommercialdimensions,spelledoutthedevelopmentpolicyfortheGovernmentandindustrydecisionmakersdealingwithbigdata.

InJanuary2012,the“WallStreetJournal”arguedthatbigdata,smartproductionandwirelessnetworktwillleadtoneweconomicprosperity[8].

InMarch2012,theUnitedStatesgovernmentreleased“BigDataResearchandDevelopmentInitiative”,whichrosesthedevelopmentandapplicationofbigdatafrombusinessconducttonationaldeploymentstrategicinordertoimprovetheabilitytoextractknowledgefromlargeandcomplexdata,tohelpsolvesomeofthenation'smostpressingchallenges.

InApril2012,“NatureBiotechnology”invitedeightbiologiststoevaluateanarticlewhichpublishedinDecember2011on“Science”titling“DetectingNovelAssociationsinLargeDataSets”inapapertitled“Findingcorrelationsinbigdata”.

InJuly2012,Gartnerreleasedthefirstdatasurveyreport“HypeCycleforBigData,2012”,whichthoughtdeeplyinbigdata[9].InChina[10],bigdataattractsasmuchattentionasitdoesaroundtheworld.BaiduusesHadooptodooff-lineprocessingsince2007.Currently,Baiduhasover10,000Hadoopservers,whichismorethanYahooandFacebook,anditplanstoreach20,000in2013.Intheseservers,80%Hadoopclustersareprocessing0totalof6TBdataeverydayonloganalysis.Tencent,TaobaoandAlipayarealsousingHadooptoestablishdatawarehouseandhandlebigdata.InApril2010,Taobaolaunchedadataminingplatform“datacube”,basedonanonehundredbillionleveldatabasenamedOceanBase,whichsupportsfor4to5milliontimesupdateoperation,includingover2billionrecords,containingmorethan2.5TBdatainoneday.InMay2010,ChinaMobileestablishedamassivedistributedsystemsandstructuredmassdatamanagementsystemonthecloud.Huaweianalyzesdatabasedonmobileterminalsandstoragemassivedatathroughthecloudtoobtainvaluableinformation.Alibabaanalyzesbusinesstransactiondatathroughbigdatatechnologytodocreditapproval.BigdatadisasterBigdataiscloselyrelatedtohumandailylife,permeatedallwalksoflife.Thenumber,sizeandcomplexityareallinsharpincreasing.

Alargeamountofdatahasbeenstoredinthedatabaseandwarehouseintypesoftext,graphics,imagesandmultimedia[11].

TheresearchfromInternationalDataCorporation

hasshownthat,asof2003humanshavecreatedatotalof5EBdata,whileintheyearof2011,theamountofdatathathadbeencopiedandproducedisexceeded1.8ZB.Itisexpectedthatby2020globaldatausagewillreach35.2ZB,whichneeds37.6billionharddrivesof1TBcapacitytostore.Ontheonehandthesedatabroadensthescopeofavailablebigdataavailableforhumantogainwisdom.Ontheotherhandthevalueofasingleunitofthedataisrapidlydeclining.Humanissubmergedbythedataoceanbutthirstyforknowledge.3.1GarbageBigdataisvoluminousanditgrowsquickly,butithasverylowdensityinvalue,whichmeansthereisalotofjunkdata[12].Thestudyontheelectron-positroncolliderhasbeenabletoshoot40millionpicturespersecond,butonlyafewthousandsareuseful.RomaniaInternetsecuritycompanyBitDefenderpointedoutthat

spamandfishinginformationinthesocialnetworkgamehasincreasedbymorethan50%.Comparedtootheronlinecommunicationenvironment,socialnetworkusersaremoreeasilytounknowinglyacceptandloadgarbageinformation.Bigdataandapplicationsarecloselyrelated,andprofessionallabelingofthedataisthebasicobjectiveofrationalanalysisandsoundjudgment.

Whetherscientificexperimentaldataorobservationdataneedtobelabeledbyexpertsinthefield.

AccordingtotheIDCstatistics,in2012only23%ofallinformationisuseful,ofwhichonly3%ofpotentiallyusefulinformationhadbeenlabeled,andtheproportionofdatawhichhadbeenanalyzedismuchless.Withthedevelopmentofmodernmeasuringtechniqueanddigitalrecordingmethod,inthefaceofhugeinformation,traditional,artificial,experienceeliminationandanalysismethodshavebecomepowerless.3.2ContaminationDatacollectedfromtherealworldiscontaminated.Moreover,asearlyas1992,theMassachusettsInstituteofTechnologyfoundthatdatacontaminationproblemsarenotisolated.Inthe50unitsandagenciesthataresampledforthesurvey,mostofthedataaccuracyislessthan95%.

Regardlessofaccesstos8atialdata,therearesomeinevitableproblemsorerrors[13][14],suchascontentsincomplete,precisionerror,dataredundancy,formatcontradictory,differenttype,structureuncertainties,differentscales,differentstandard,outdated,errorexception,dynamicchangeandlocalsparse.Moreovereachissuehasanumberofcauses.Forexample,thenoisecanbeperiodicnoise,stripenoise,isolatednoiseandrandomnoise.Further,thesedataareoftenaffectedbygrosserrors,systemerrorsandrandomerrorsindividuallyorcollaboratively.

Itisboundtodamagetheexpecteddataaccuracyifthesethreekindsoferrorcannotbecorrectlyfoundandeliminatedintheadjustment.3.3DifficulttouseDataisnotonlycontaminated,butalsodifficulttouse.Theproduction,transmission,replicationandaccumulationofdatahavegonefarbeyondpeople'scapacityforanalyzing,understandingandimplementing.Duetothelargeamountof“bigdata”,itisdifficulttocollect,store,search,share,analyzeandmaterialize.

Commercialimageprocessingsoftware(ERDAS,IMAGINE,PCI,ENVI,etc.)aredifficulttocompletethefollowingmission:mixpixel,imagematchautomatically,targetextractautomatically,andotherautomaticprocessingmissionbecausethelackofnewtheoriesandmethods.Anewspaperpublishedthesamearticleofthesameauthorontwodifferentpagesof“legalcommunity”and“youthtopics”.AnothernewspaperpublishedthreearticlesintheEditionof“homeappliances”,“lifestyle”and“scienceandtechnology”,alltocompareamongVCD,CVDandDVDonthesameday,andgotthreedifferentconclusions,buttheeditordidnotevenrealizeit.Overtime,allwalksoflifearesubmergedbycontaminateddatagarbage,andthenitcouldleadthebigdatainto“garbagein,garbageout”,andthe“bigdata”becomestheuseless“biggarbage”.Now,usefuldataisburied,andimpliedvalueisblankedinbigdata.

Onsuchapredicament,followingproblemsarethebottlenecksforbigdataresearchtobreakthrough:howtounderstandthespatialdata,howtoextractinformationfromthedata,howtoturndataintoknowledgecanbeavailable,andfinallyhowtorealizethevalueofdata.ThevalueofdataBigdataiscollectedfromnumerousandinterconnectedsources.Realusefulnessisitsmaximumvalue.Thegenerallyacceptedruleofbigdatais“decisionondata”.Thefirstprerequisiteistokeepdataalwaysusefulandactivated.Theultimatevalueofbigdataistogainhumanintelligence.4.1OverallcognitiveoriginalappearanceBigdataprovidesanunprecedentedopportunitytoobservetherealworldinafullviewratherthanpartialsamples.Withoutbigdata,probabilitystatisticscanonlybeproducedbasedonrandomsamplingfromtherealworld,becausespacedataisconstrainedbycollection,storage,computingandtransmission.Liketheproverbialblindmengraspinganelephantcanonlytakeapartforthewhole,thereisonlyalimitedview.Incompletedatasamplingandsampledatadispersionmakeitdifficulttounderstandtheoveralltrendsortonoticetheabnormalchanges.4.2BasicresourcesMcKinseybelievesthatdataisthebasicresource,andcanbecomparedwithphysicalassets,humancapital,createsignificantvaluefortheworldeconomy,improvetheproductivityandcompetitivenessoftheenterprisesandthepublicsector,andcreatealargenumberofeconomicsurplusforconsumers.In2011,theWorldEconomicForumcalledbigdataasnewwealth.In2012,theDavosForum“BigData,BigImpact”treateddataaseconomicassetlikecurrencyorgold.In2012,Gartnerbelievesthat“Bigdataisbigmoney”.

TheU.S.governmentconsidersbigdataas“newoil”relatedtothecountry'seconomicrestructuringandindustrialupgrading[3].DataminingDataminingreferstothebasictechnologiestorealizethevalueofbigdata,relocatedataassets,anduseiteffectively.Spatialdataminingcanbeusedtoextractinformationfromdata,mineknowledgefrominformation,extractdataintelligenceinknowledge,improvetheabilityofself-learning,self-feedbackadaptation,finallyrealizehuman-machineintelligence.5.1BasicbigdatatechnologyThebasictechniquesofbigdataincludedatacollection,storage,processing,expression,andqualityevaluation.Bigdatacanbegeneratedinmobiledevices,trackingsystems,radiofrequencyidentificationdevices(RFIDs),sensornetworks,socialnetworking,Internetsearch,automaticrecordingsystems,videoarchives,e-commerce,aswellastheprocessinanalyzingthosedata.Bigdatastoragetechnologyisthebasisfordatamining.Itisdesignedtomeetthegrowingneedfordatastorage,whichaimstoprovidescalability,highreliability,excellentperformancedatastorage,access,andmanagementsolution,suchasdistributeddatastorage,multiplelevelscaching,loadbalancing,fault-tolerantmechanisms.Conventionalmethodsarenotadequateforthesemissions.Itneedstoestablishalargeplatformfordatathroughsoftware,toprovideplacestostoreandinterfacetoaccess.

Bigdataprocessingistoimplementthetransitions:fromdatatoinformation,frominformationtoknowledgeandfromknowledgetowisdom.

Bigdataexpressiontechnologyisdesignedtorepresentthedatainaclearandeffectivewaythatrevealsmeaningfulinformationtotheuser,orprovidetheuserwithanewperspectiveofview.Bigdataexpressiontechnologyincludesdigitalelevationmodels,digitalterrainmodels,flatmaps,three-dimensionalmaps,anddigitalcitymaps.Bigdataqualityassessmenttechnologyisaimedtoavoidtheriskofbigdatacollectingandhigh-densitymeasuring.Thetechnologyincludeslogicalassessmentmethod,exceptionvaluebasedassessmentmethod,andaccountingbasedassessmentmethod.5.2DiscoveryknowledgeKnowledgediscoveryisthetechnologythatusesdataminingmethodtoextractpreviouslyunknown,potentiallyuseful,andultimatelycomprehensiblerules.Itisalsoaprocessofgradualsublimationfromdatatoinformation,andtoknowledge,step-by-step.Dataminingsystemsaimstomakedatagraduallysummarizedintoknowledge.Throughtheintegrationofdata,itcandeeplyextractknowledge.Byusingsuchnewknowledge,datacanbeprocessedinrealtimeinordertounderstandandapplythedata,tomakeintelligentjudgmentsandwell-informeddecisions.Knowledgecanbeself-learning,self-enhance,universal,andeasilyrecognized.Itcouldserveasabasisfordecisionsupport.Ifbusinessestakefulladvantageofknowledge,itwillbemorepreciseanddynamicforhumanstolearn,work,life,andachievewisdomstate.Itwillhelptoimproveresourceutilizationandproductivitylevel.Moreover,itwillalsohelptorespondtotheeconomiccrisis,theenergycrisis,thedeteriorationoftheenvironmentandmanyotherglobalissues.5.3ExtractiondataintelligenceDataintelligenceistheabilitytoobtainamoreinnovative,systematicandcomprehensiveknowledgetosolveaparticularproblemthroughanin-depthanalysisofthecollecteddata.Itisanabilitytounderstandandsolveproblemsfast,flexiblyandcorrectly.Spatialdataintelligenthasthreefeatures:morethoroughlyperception,moreextensiveinteroperability,anddeeperintelligence.

Thethreefeaturesareaimedtogetbiggerandmorecomprehensivedata,toshareandco-operatedataviatheInternet,tododataanalysisanddataminingbyvarietyofadvancedtechniques,andtoconstituteahierarchyofspatialdataintelligences(Fig1).Figure1.

ThehierarchyofspatialdataintelligencesBigdataintelligencedoesnotrefertosimpleoverlaydifferentdataminingtechniques,butareasonablestructureofindustry-orientedorganization,goodrunner,andpowerfulwisdomsystem.Themorereasonableindustrystructurebecome,thesmallerinternalfrictiongot,thegreatereffectivenessgot,andthehigherwisdomsystemgot.Everytimewhenapersoninteractingwiththedatahe/shebecomesmoreefficientandmoreproductive,whichmeansitformsabetterwaytoanalyze,summarize,andcalculate.Throughtheconsolidationandanalysisoftrans-regional,trans-sectordata,withknowledgeappliedinspecificindustry,specificscenesandspecificsolution,bigdataintelligencecansupportdecision-makingandactioninabetterway.

Morein-depthdataintelligenceistocreatenewvalueofdata.Ontheonehand,whenmakingfulluseofspatialdataknowledgeinallwalksoflife,itcanproducesecondaryknowledge.Inordertoformaminingmechanismtomineknowledgeinknowledge,itneedstobringprimaryknowledgetogethertoformanintelligentformofexpression.Ultimately,thedestinationknowledgecanbeachieved.Ontheotherhand,basedonageneralindustrialorsocio-ecologicalsystem,itcanredefinetheinteractivemodeofgovernment,companiesandindividuals,sothatitimprovestheinteractionclarity,efficiency,flexibilityandresponsespeed.Itchangesfromthetraditionalsingledimensionsuchas:productionconsumption,managementbemanagement,orplanningexecution,toanewmulti-dimensionalcollaborativerelationship.Inthisnewrelationship,bothindividualsandorganizationscanfreelycontributeandgetinformationandexpertiseaccuratelyandtimely.Thisnewrelationshipexertsapositiveinfluenceoneachothertoreachsmartrunningmacro-effects.EffectivenessWhenwepossessthenecessaryknowledgeandabilitytocontrolit,thedatabecomesourvaluableassetthatleadstomarketdominationandhugeeconomicreturns.

Bigdatatechnologyprovidersusetechnologyforusersprocessingstructured,semi-structuredandunstructureddata.BigdataapplicationsareincreasinglyInternetubiquitous,richinterfaced,andfragmented.Itisaverticalintegrationintheapplicationindustry,therefore,businessthatisclosertoend-users,tendstohavealargerinfluenceintheindustrychain.MorganStanley'sreportinsiststhat“BigDataissoontobecomeAnyData[15]”,Inordertowinthefuture,therationalchoiceisthat“givingcustomersthetechnologiestheyneedtostoreandanalyze‘any’dataset-anytypeofdata,anysizeofdata,foranytypeofuser,andinanytimeframe.”ConclusionThedevelopmentofbigdataextendsthescopeofhumanactivities.Itdemandsproperattentionfromacademia,industryandgovernment.Theworldhasbeencooperatingandintegratingonaglobalscale.Humanisenforcedtochangemodefromthelocaltotheglobalintheireverydaylifeandwork.Itredefinestherelationshipamongindividuals,businesses,organizations,gov