IBM Big Data Plat Overview.ppt

ibmbigdataplatformoverview,martinpavlk+420731435691martin_pavlik,bigdataisahottopicbecausetechnologymakesitpossibletoanalyzeallavailabledata,costeffectivelymanageandanalyzeallavailabledatainitsnativeformunstructured,structured,streaming,erp,crm,rfid,website,networkswitches,socialmedia,billing,bigdataisnotjusthadoop,manage&storehugevolumeofanydata,hadoopfilesystemmapreduce,managestreamingdata,streamcomputing,analyzeunstructureddata,textanalyticsengine,datawarehousing,structureandcontroldata,integrateandgovernalldatasources,integration,dataquality,security,lifecyclemanagement,mdm,understandandnavigatefederatedbigdatasources,federateddiscoveryandnavigation,business-centricbigdataenablesyoutostartwithacriticalbusinesspainandexpandthefoundationforfuturerequirements,“bigdata”isntjustatechnologyitsabusinessstrategyforcapitalizingoninformationresourcesgettingstartediscrucialsuccessateachentrypointisacceleratedbyproductswithinthebigdataplatformbuildthefoundationforfuturerequirementsbyexpandingfurtherintothebigdataplatform,1unlockbigdata,2analyzerawdata,mergingthetraditionalandbigdataapproaches,itstructuresthedatatoanswerthatquestion,itdeliversaplatformtoenablecreativediscovery,businessexploreswhatquestionscouldbeasked,businessusersdeterminewhatquestiontoask,monthlysalesreportsprofitabilityanalysiscustomersurveys,brandsentimentproductstrategymaximumassetutilization,bigdataapproachiterative&exploratoryanalysis,traditionalapproachstructured&repeatableanalysis,infospherebiginsightsismorethanjusthadoop,hadoop,open-sourcesoftwareframeworkfromapacheinspiredbygooglemapreducegfs(googlefilesystem),hdfsmap/reduce,infospherebiginsights,platformforvolume,variety,velocityenhancedhadoopfoundationanalyticstextanalytics&toolingapplicationacceleratorsusabilitywebconsolespreadsheet-styletoolready-made“apps”enterpriseclassstorage,security,clustermanagementintegrationconnectivitytonetezza,db2,jdbcdatabases,etc,apachehadoop,basicedition,enterpriseedition,licensed,applicationacceleratorspre-builtapplicationstextanalyticsspreadsheet-styletoolrdbms,warehouseconnectivityadministrativetools,securityeclipsedevelopmenttoolsperformanceenhancements.,freedownloadintegratedinstallonlineinfocenterbigdatauniv.,breadthofcapabilities,enterpriseclass,canrunalsoontopof,spreadsheet-styleanalysis,web-basedanalysisandvisualizationspreadsheet-likeinterfacedefineandmanagelongrunningdatacollectionjobsanalyzecontentofthetextonthepagesthathavebeenretrieved,buildabigdataprogrammapreduceexample,eclipsetoolsforjaql,hive,pigjavamapreduce,bigsheetsplug-ins,textanalytics,etc.,jaqlibmsprogramminglanguageinhadoopworld,jaqlisacompletesolutionsenvironmentsupportingallotherbiginsightscomponents,integrationpointforvariousanalyticstextanalyticsstatisticalanalysismachinelearningad-hocanalysisintegrationpointforvariousdatasourceslocalanddistributedfilesystemsnosqldatabasescontentrepositoriesrelationalsources(warehouses,operationaldatabases),biginsightstextanalytics,statisticalanalysis(rmodule),machinelearning(systemml),ad-hocanalysis(bigsheets),(integration)db2,netezza,streams,jaql,jaqli/o,jaqlcoreoperators,jaqlmodules,dfs,nosql,rdbms,filesystem,biginsights,datawarehouse,traditionalanalytictools,bigdataanalyticapplications,filtertransformaggregate,biginsightsandthedatawarehouse,3simplifyyourwarehouse,analyst,it,ineedtoevaluatethepossiblerelationshipbetweenclientsalaryandoverdrafts,ok.wehavetoevaluatealotofstatistics,setthecorrectdbindexesanddbpartitioning.itwilltakeus5days.,analyst,it,great.thanksalot.imgoingtochecktheresults.,done.youcanrunyouranalyticalquery.,after5days.,analyst,it,great.icanseeheresomenicecorrelations.nowineedtolookatitfromthedifferentperspective.,after10minutes.,ohhh,welcomedearfriend.understand.so,its.another5daysofourwork,noooo!itsnotpossibletoworkhere!,andnowwithnetezza.,analyst,it,ineedtoevaluatethepossiblerelationshipbetweenclientsalaryandoverdrafts.iwillusenetezza.,analyst,it,great.icanseeheresomenicecorrelations.nowineedtolookatitfromthedifferentperspective.withnetezzaicanrunthequeryimmediately.theresponsewillbeinthesametime,after12minutes.,itcandosomethingelsemuchmoreuseful,gotoviewheaderandfootertochangethisfootertexttotheeventtitle,23,built-inexpertisemakesthisassimpleasanappliance,dedicateddeviceoptimizedforpurposecompletesolutionfastinstallationveryeasyoperationstandardinterfaceslowcost,ibmnetezzawasrenamedtoibmpuredatasystemforanalytics,inoctober2012,netezzagenesisint-mobilecz,proof-of-conceptprojectnewenterprisedatawarehouseplatformselectioncomparisonofexistingandotherplatformsselectioncriteriaperformanceoperationalsavings.andthewinnerwas:netezza,netezzagenesisint-mobilecz,expectationssignificantresponseimprovement:fasterplatformmeansbetterreportsresponsedirectdataavailabilityhighertrustindata,oneversionoftruthaggregationreductionanyattributeavailableoperationalbenefitsstoragesavings(nodatareplicas)administrationcostsreduction(dba)infrastructuresimplificationlowerenvironmentcomplexity,netezzagenesisint-mobilecz,projectimplementationedwplatformmigrationnetezzaplatformimplementationetlgraphs/processesredesignbifront-endtoolmigrationsapbusinessobjectimplementationallreportsredesignmainintegrationpartner:t-systemcz,netezzagenesisint-mobilecz,actualstatusallrelevantetlprocecessingredesignedactualparallelruntooriginalandnetezzaplatformfinishednetezzaasonlyprimaryplatform,responsetimemassivelyimproved,realnetezzaexperiencefromt-mobileczechrep.,4reducecostswithhadoop,biginsightsandthedatawarehouse,biginsights,query-readyarchivefor“cold”warehousedata,datawarehouse,bigdataanalyticapplications,traditionalanalytictools,fromcognosbiviahivejdbc,biginsightsconnectivitytodbms/warehouse,netezza,biginsights,jdbcdbms,db2luw,iwwithdpf,biginsightsdrivesrdbmsworkdbmsdrivesbiginsightswork,cognosbusinessintelligencebigdataarchitecture,loadthroughudf,netezza,rdbms,cognosinsight,application(map-reduce,lucene,systemt),storage(hbase,hdfs,gpfs),querymethods(jaql,pig,hive),csv,infospherebiginsights,restviahttp,cognosbiserver,textanalytics,restapi,explore&analyze,report&act,visitthebiginsightswikiforalinktoanarticleoncognosandbiginsights,application,sqlinterfaceengine,infospherebiginsights,hivetables,hbasetables,csvfiles,datasources,future:thesqlinterface.,richsqlquerycapabilitiessql92and2011featurescorrelatedsubquerieswindowedaggregatessqlaccesstoalldatastoredininfospherebiginsightsrobustjdbc/odbcsupporttakeadvantageofkeyfeaturesofeachdatasourceleveragemapreduceparallelismorachievinglow-latency,5analyzestreamingdata,whyandwhentouseinfospherestreams?,atleast2criteriafromthelistbellowshouldbefulfilled,applicationsneedingon-flyprocessing,filteringandanalyzingstreamingdata,streamsandwarehouse:complementary,high,med,low,low,med,high,latency,1pb,b,kb,gb,10gb,1