文本文稿分析

1、August 2013FlexPod Select with HadoopTechnical Customer PresentationThe Big Data Analytics Gap2Existing SkillsApplicationsDataGrowthSource: IDCLost Value“Big data” refers to datasets whose size is beyond the ability of current tools to capture, store, manage, and analyzeChallenges with Hadoop in Ent

2、erprise3Operations ImplementationRequires three copies of data, larger footprint, and more storageLimited flexibility; storage and servers tied together affects scalability Low cluster efficiency, higher network congestionNameNode is a single point of failureSlow recovery from disk drive failureExpe

3、nsive process to replace failed disks onlineMost common Hadoop support issue is disk drive failureAvailabilityNeed to keep up with fast-paced patches, projects of open source platformNeed to decide on distribution of Hadoop Skills are not commonIntegration with existing IT infrastructure can be diff

4、icultTuning expertise needed to make Hadoop perform optimallyBig Data Opportunities4Fraud detection and preventionAnti-money-laundering Risk managementSupply-chain optimization Defect tracking Root-cause analysisRFID correlationLaw enforcementCounterterrorismDrug developmentPatient recordsEvidence-b

5、ased medicineFinancial ServicesManufacturingHealthcareGovernmentBig Data: Problems Being Solved TodayFinancial InstitutionsTick data analysisTrend analysisRisk assessmentCross-domain correlationsHealth CareDrug effaceDisease pattern recognition and progressionFraud detectionClaim automationGovernmen

6、t (FBI,CIA,DOD,DOE,HLS,IRS,SEC)Internet threat detectionPattern recognitionImage analysis Fraud and waste mitigationConsumer and Commercial SpaceSocial media and product sentiment correlationConsumer analysisAdvertising affinity correlationTelemetry and quality analysis5Changing the Way Business Is

7、ConductedValue Proposition6Some Problems Require an Enterprise-Class Hadoop SolutionEnterprise Class HadoopPackaged ready-to-deploy modular Hadoop cluster The data has intrinsic $ value Usable capacity must expand faster than compute Higher storage performanceReal human consequences if the system fa

8、ils (threats, treatments, financial losses)System has to allow for asymmetric growthWhite Box HadoopValues associated with early adopters of Hadoop Social media space Contributors to Apache Strong bias to JBODSkeptical of all vendorsEnterprise-Class HadoopPackaged ready-to-deploy modular Compute- an

9、d memory-intensive Hadoop cluster Compute-intensive applicationsTick data analysisExtremely tight service-level expectations Severe financial consequences if the analytic run is lateEnterprise Class HadoopBounded compute algorithm and memory-intensive Hadoop cluster Compute-intensive applicationsAdd

10、itional CPUs do not improve run timeExtremely tight service-level expectations Severe financial consequences if the analytic run is lateNeed for deeper storage per DataNodeCompute PowerStorage Capacity7FlexPod FamilyFlexPod Converged Infrastructure Prevalidated, Flexible, Unified Platform8Cisco Unif

11、ied Computing SystemProgrammable infrastructureAbstraction of bare metal server elementsOpen API for automationOrchestration through industry-standard toolsUnified managementPolicy-based automation and service profilesSelf-integrating componentsUnified fabricVirtualization awarenessScalability witho

12、ut complexityHigh-performance I/ONetApp Storage Systems: FAS with Clustered Data ONTAP ; E-Series with Dedicated ApplicationsUnified storageVirtualize storage into pools Nondisruptive data motionIntelligent data managementBuilt-in storage efficiencyIntegrated data protectionHigh performance, dedicat

13、ed workloadsDensity, scale for big dataThroughput for video, media, HPCFlexPod Converged Infrastructure Family50 PBStorage PoolNetwork PoolCompute Pool10,000scoresAppAppAppAppAppAppEnterprise/Service ProviderMSB/Branch OfficeDedicatedBasic Cisco UCS ServersAppDistinct ArchitecturesDistinct Architect

14、uresFlexPod ExpressFlexPod Data CenterFlexPod SelectTestedFull ValidationTested or full validationFAS2220, 2240 (currently 7-Mode) FAS and Data ONTAP (clustered and 7-Mode)Any: E-Series, FAS, FlashRayCisco UCS C-Series, Nexus 3K Cisco UCS, Nexus, UCS FI, UCS Manager Cisco UCS, Nexus, Catalyst, MDS6,

15、000+ commercial resellers800 FlexPod; 44+ premium partnersSpecialized and vertical resellersTwo fixed pod sizesFlexible pod sizesReference architecture and/or designs1GbE fabric10GbE fabricDirect, FC, and 10GbE Cisco UCS Director, VMware, and MicrosoftFlexPod validated management and ecosystemApplic

16、ation-based managementFile SystemMassively scalable shared virtual data center infrastructureBig data analytics, scientific, HPCFor smaller, less-dynamic requirements and VAR velocityDirect or FabricAppApp9FlexPod Data Center Versus FlexPod Select 10FlexPod Data Center (Classic)Architecture includes

17、 Cisco UCS B- or C-Series servers managed by UCS Manager and Fabric InterconnectNetApp FAS storage connected to N5K or N7K access layer (which is connected to UCS Fabric Interconnect)Storage connectivity through fabricData protocol is FC/FCoE or Ethernet-based iSCSI/NFS/CIFSTargeted for data center

18、enterprise applications such as Microsoft apps, VDI, SAP, Oracle, and so onFlexPod Select Architecture includes Cisco UCS C-Series servers only, managed by UCS Manager and Fabric InterconnectPrimarily NetApp E-Series storage for datastore E-series connected to Cisco UCS C-Series servers directly (SA

19、S attached)Storage connectivity through direct-attach to serverData protocol is SASTargeted for Hadoop and big data workloads such as Cloudera and HortonworksNetApp FAS is used for NameNode store (not for datastore)Extending FlexPod to Big Data11Cisco UCS Rack-Mount ServersCisco UCS Blade ServersExt

20、endable to multiple data center implementations for disaster recovery and business continuityCisco UCS ManagerDeploy, Manage, MonitorCisco Tidal Enterprise SchedulerHadoop ConnectorsBig DataEcosystemNetApp FASEnterprise ApplicationsAvailabilityBackupNetApp SnapshotFlexPod Select with Cloudera Enterp

21、rise-grade Hadoop management applicationEnd-to-end administration in a single toolProven at scale to work in production environmentsBuilt-in intelligence, best practices, real-time queriesCisco Validated DesignMasterExpansion Management Software & Technical Support (Included)CDHCMCloudera ManagerCSC

22、loudera SupportOSSApache Hadoop & Open Source Software12FlexPod Select with Hortonworks Deploy seamlessly across multiple operating systems, clouds, virtual platformsThe only100% open source and complete distributionEnterprise grade, proven and tested at scaleEcosystem endorsed for interoperabilityM

23、asterExpansion13Solution Architecture14Solution ArchitectureData ONTAPFAS222015FlexPod Select with Hadoop ArchitectureE5460 storage5 disk trays60 NL-SAS drives6Gb/s SAS (direct attached to servers)FAS2220-2 Data ONTAP 8.1.2 7-ModeCopy of HDFS metadataNFS served to NameNode and secondary NameNodeFAS

24、222016Front View of Master and Expansion RacksMasterExpansionE5460 storage arraysData and Task NodesInfrastructure,NameNode,secondary NameNode,JobTrackerCisco UCS 6296UP (FI)Cisco Nexus 2232PP (FEX)FAS222017Back View of Master and Expansion RacksMasterExpansionE5460 Storage ArraysData and Task Nodes

25、Infrastructure,NameNode,secondary NameNode,JobTrackerCisco UCS 6296UP (FI)Cisco Nexus 2232PP (FEX)FAS222018E5460 Storage Layout for HDFSTwo 7-disk RAID groups with two LUNs per nodeDedicated set of disks per DataNodeShared-nothing architectureSpare disks shared globallyRAID 519FAS2220 Design for Had

26、oop NameNodeMetadata protection through NFSHighly reliable, proven NetApp NFSPrimary and secondary NameNodes have NFS access to FSImage dataQuick turnaround to rebuild NameNode in case of failurePrimary NameNodeSecondary NameNodeNFS 10GbENFS 10GbEFAS222020Architecture OverviewFlexPod Select with Had

27、oop21What Is FlexPod Select with Hadoop?The Cisco Platform for NetApp Open Solution for Hadoop is an implementation of the NetApp Open Solution reference design using the Cisco Platform Architecture for Big Data.Like the NetApp Open Solution for Hadoop v1.0, the Hadoop Rack is based on building bloc

28、ks of Hadoop data and compute servers using NetApp E5460 storage arrays instead of internal local hard-disk drives.The basic building block is four servers for each E5460 storage array.22FlexPod Select with Hadoop ConfigurationMaster rackInfrastructure node; Hadoop NameNode, secondary NameNode, and

29、JobTracker node3 E5460 storage arrays supporting 12 Hadoop data and compute nodes12 Hadoop DataNode and TaskNode serversLocal DNS/NTP serverFAS2240-2 storage array for Hadoop NameNode metadata backupIn-rack 10GbE and 1GbE network infrastructureExpansion rackFour E5460 storage arrays supporting 16 Ha

30、doop DataNodes and TaskNodes16 Hadoop data and compute node serversIn-rack 10GbE and 1GbE network infrastructureNote: Each Hadoop cluster can have only one master rack, but it may have one or more expansion racks.23Why Have a Local DNS Server?BenefitsEliminates the need to modify other enterprise or

31、 data center DNS serversIs easily integrated into enterprise DNS servicesHelps isolate the Hadoop cluster from other enterprise network namespacesServices provided for the Hadoop clusterBoth forward and reverse host name and IP address resolution for the Hadoop clusterA platform for a running NTP ti

32、me service so that all Hadoop cluster resources have a good synchronized time sourceA local copy of NetApp SANtricity Storage Manager for administration and monitoring of the E5460 storage arraysA platform for automated storage provisioning scripts during the racks initial build Possible future serv

33、er servicesService to rebuild failed servers with correct OS version and required packages, such as Kickstart and PuppetHadoop management software24FlexPod Select with Hadoop Networking25Component10GbE ConnectionsGbE ConnectionsE5460 arraysNoneConnection to each of two controllers in the arrayFAS222

34、0Four 10GbE portsConnection to baseboard management Controller and remote LAN module management portNodesTwo 10GbE portsInfrastructure serverTwo 10GbE portsIn-Cabinet Network AdvantagesProvides better performance with 10GbE than traditional GbE Hadoop networksDedicates 10GbE bandwidth to where it is

35、 requiredOffers ease of Hadoop network administration:Most Hadoop network administration is performed on the 10GbE network in the master rackComponent (management) 1GbE administration is performed on the 1GbE switch in the local rack of the affected componentsIsolates Hadoop network traffic from oth

36、er enterprise network trafficMinimizes network cabling to external network infrastructureFacilitates rack network securityUses VLANs to separate administration traffic from Hadoop cluster traffic26Storage System Components27Two Main Storage ComponentsE-Series E5460 storage system FAS2220 storage systemFAS2220,16x1 TBE5460, 4U 60 x3TB/4TBDense Innovative Design for Hadoop4U, sixty 3.5 drive high-density SAS enclosure (up to 240TB via 4TB drives)Five12-drive horizontal drawers = 60 drivesTwo E5400 controllers and SAS HICsOnline front serviceabl