VCS-培训教材(中文)

VERITASClusterServer,CONTENTS,VCS预备知识VCS的基本概念和术语VCS的管理集群服务VCS常见问题的解决总结,Page2,VCS预备知识,Page4,MetropolitanHADisasterRecovery(overSAN,MANorLAN),WideAreaDisasterRecovery,VCS的几个常用场景,LAN,LocalClustering,MAN,WAN,ClusterServer,ClusterServer,VolumeManager,ClusterServer,VolumeManager,VolumeReplicator,GlobalClusterManager,Page5,Clustering:ApplicationandDatabaseFailover数据库和应用的失效转移,AClusterViewofApplications,A“servicegroup”isacollectionofresourcesthatmonitorthestatusofanapplication（服务组是各种监控应用状态的资源的集合）Applicationfailoveriscontrolledbytheservicegroup（应用的失效转移是由服务组来控制的）,1.,1.,2.,2.,3.,4.,5.,Page6,Active/PassiveClustering（主备方式）“asymmetricconfiguration”（非对称配置）,Primaryserverhostsapplication（主机提供服务,PrimaryserverFAILS,Secondaryserverhostsprimaryapplication,备机处于等待状态，一旦主机发生故障，接管服务）,Page7,Active/ActiveClustering（互备方式）“symmetricconfiguration”（对称配置）,Primaryserverhostsprimaryapplication,PrimaryserverFAILS,Secondaryserverhostsbothprimaryandsecondaryapplications,Secondaryserverhostssecondaryapplication两个节点提供不同的服务，互相备用，当一个节点故障，服务马上有第二个节点接管服务,Page8,VCS的基本概念和术语,集群,SCSIJBODS,Severalnetworkedsystems几个节点Sharedstorage共享存储Singleadministrativeentity单个管理节点Peermonitoring相互监控,FibreSwitches,Page10,systems系统,Membersofacluster集群的一个成员Referredtoasnodes也称之为节点Containcopiesof:包括如下内容Communicationprotocolconfigurationfiles通信协议的配置文件VCSconfigurationfilesVCS的配置文件VCSlibrariesanddirectoriesVCS的安装目录VCSscriptsanddaemonsVCS的脚本和后台程序Shareasingledynamicclusterconfiguration共享一个动态的集群配置Provideapplicationservices提供应用的服务,Page11,ServiceGroups服务组,Aservicegroupisarelatedcollectionofresources.服务组是资源的一个集合Resourcesinaservicegroupmustbeavailabletothesystem.服务组中的资源在系统中必须是可用的Resourcesandservicegroupshaveinterdependencies.服务组和资源存在相互依赖关系,NFSServiceGroup,NFS,IP,Disk,Mount,Share,NIC,Page12,ServiceGroupTypes服务组的类型,Failover失效转移Canbepartiallyorfullyonlineononlyoneserveratatime同一时间只能在一台机器上运行VCScontrolsstoppingandrestartingtheservicegroupwhencomponentsfail当服务组某个资源出错时，VCS控制它的停止和重启Parallel并行Canbepartiallyorfullyonlineonmultipleserverssimultaneously可以同时在多台机器上运行Examples:OracleParallelServerWeb,FTPservers,Page13,Resources资源,VCSobjectsthatcorrespondtohardwareorsoftwarecomponents包括软件和硬件组件MonitoredandcontrolledbyVCS通过VCS来监控和控制Classifiedbytype通过资源类型分类Identifiedbyuniquenamesandattributes通过唯一的名称和属性来标识Candependonotherresourceswithinthesameservicegroup在同一服务组中可依赖其他资源,Page14,ResourceTypes资源类型,Generaldescriptionoftheattributesofaresource通常描述一种资源的属性ExampleMountresourcetypeattributes:例如mount资源类型的属性MountPoint挂载点BlockDevice挂载设备Otherexampleresourcetypes:其他类型的资源Disk磁盘Share共享IP浮动IPNIC网卡,Page15,Agents代理,Processesthatcontrolresources控制资源的程序Oneagentperresourcetype每种类型的资源对应一个代理Agentcontrolsallresourcesofthattype.一个代理控制对应类型的所有资源AgentscanbeaddedintoVCSagentframework.用户可以加入自己的代理到VCS的框架中,Page16,Dependencies依赖关系,Resourcescandependonotherresources.资源可以依赖其他资源Parentresourcesdependonchildresources.父资源依赖子资源Servicegroupscandependonotherservicegroups.服务组可以依赖其他服务组Resourcetypescandependonotherresourcetypes.资源类型之间也存在依赖，比如IP类型必须依赖NIC类型Rulesgovernservicegroupandresourcedependencies.资源和服务组之间的依赖关系由规则管理Nocyclicdependenciesareallowed.不允许出现循环依赖,Mount,Disk,(Parent),(Child),Page17,PrivateNetwork私有网络,Minimumtwocommunicationchannelswithseparateinfrastructure:至少需要两条独立的通信链路MultipleNICs(notjustports)多块网卡Separatehubs,ifused独立的hubHeartbeatcommunicationdetermineswhichsystemsaremembersofthecluster.心跳之间的通信决定哪些系统是集群的成员Clusterconfigurationbroadcastupdatesclustersystemswithstatusofeachresourceandservicegroup.集群中的资源和服务组的状态信息通过广播更新到各个节点,Page18,LowLatencyTransport(LLT)低时延传输协议,Providesfast,kernel-to-kernelcommunications提供快速，内核到内核的通信IsconnectionorientedIsnotroutable不需要路由UsesDataLinkProviderInterface(DLPI)overEthernet使用以太网的链路层,Page19,GroupMembershipServices/AtomicBroadcast(GAB),Managesclustermembership管理集群成员Maintainsclusterstate维护集群状态Usesbroadcasts使用广播RunsinkerneloverLowLatencyTransport(LLT)运行在llt之上,Page20,VCSEngine(had)VCS的引擎,Maintainsconfigurationandstateinformationforallclusterresources维护整个集群的所有资源的配置和状态信息UsesGABtocommunicateamongclustersystems通过gab与集群的其他成员通信Ismonitoredbyhashadowprocess由后台进程hashadow来监控,hashadow,SystemA,SystemB,LLT,LLT,Hardware,Kernel,PrivateNetwork,had,had,Page21,Page22,VCSArchitecture总体架构,SystemA,SystemB,SharedClusterConfigurationinMemory,Hardware,Kernel,Resources,Agents,Mount,hashadow,hashadow,/v,Disk,c1d0t0s0,hme0,NIC,IP,,had,/v,Disk,c1d0t0s0,hme0,NIC,IP,had,Mount,LLT,LLT,GAB,GAB,VCS管理集群服务,ClusterConfiguration集群配置,Page24,StartingVCS启动VCS,main.cf,ClusterConf,PrivateNetwork,System2,System3,System1,Page25,StartingVCS:SecondSystem,main.cf,hadhashadow,PrivateNetwork,hadhashadow,System2,System3,ClusterConf,ClusterConf,System1,Page26,StartingVCS:ThirdSystem,main.cf,hadhashadow,hadhashadow,main.cf,main.cf,hadhashadow,System1,System2,System3,SharedClusterConfigurationinMemory,PrivateNetwork,Page27,StoppingVCS停止VCS,Page28,ThehastopCommand停止命令,ThehastopcommandstopstheVCSengine.Syntax:hastopoptionarg-optionOptions:-local-force|-evacuate-syssys_name-force|-evacuate-all-forceExample:hastop-systrain4-evacuate,Page29,DisplayingClusterStatus显示集群的状态,ThehastatusCommandDisplaysstatusofitemsinthecluster.Syntax:hastatus-optionarg-optionargOptions:-groupservice_group-summaryExample:hastatus-groupOracleSG,Page30,ProtectingtheClusterConfiguration保护集群的配置,Clusterconfigurationopened;.stalefilecreatedResourcesaddedtoclusterconfigurationinmemory;main.cfoutofsyncwithmemoryconfigurationChangessavedtodisk;.staleremoved,haconf-makerw,ClusterConf,haresadd,haconfdump-makero,main.cf,main.cf,.stale,Page31,OpeningandSavingtheClusterConfiguration打开和保存集群配置,Thehaconfcommandopens,closes,andsavestheclusterconfiguration.Syntax:haconfoption-optionOptions:-makerwOpensconfiguration-dumpSavesconfiguration-dumpmakeroSavesandclosesconfigurationExample:haconf-dump-makero,Page32,StartingVCSwithaStaleConfiguration,main.cf,hadhashadow,PrivateNetwork,hastart,hadhashadow,System2,System3,main.cf,.stale,main.cf,Page33,Page34,ForcingVCStoStartontheLocalSystem,System1,main.cf,PrivateNetwork,hastart-force,hadhashadow,System2,System3,main.cf,.stale,ClusterConf,main.cf,ForcingaSystemtoStart,Page35,ThehasysCommand,AltersorqueriesstateofhadSyntax:hasysoptionargOptions:-forcesystem_name-list-displaysystem_name-deletesystem_name-addsystem_nameExample:hasys-forcetrain11,Page36,PropagatingaSpecificConfiguration配置文件的传播,StopVCSonallsystemsintheclusterandleaveapplicationsrunning:hastop-allforceStartVCSstaleonallothersystems:hastartstaleThe-staleoptioncausesthesesystemstowaituntilarunningconfigurationisavailablefromwhichtheycanbuild.StartVCSonthesystemwiththemain.cfthatyouarepropagating:hastart,Page37,SummaryofStartOptions启动总结,Thehastartcommandstartsthehadandhashadowdaemons.Syntax:hastart-optionOptions:-stale-forceExample:hastart-force,Page38,ValidatingtheClusterConfiguration验证集群配置,Thehacfutilitychecksthesyntaxofthemain.cffile.Syntax:hacf-verifyconfig_directoryExample:hacf-verify/etc/VRTSvcs/conf/config,Page39,ModifyingClusterAttributes修改集群属性,Thehacluscommandisusedtoviewandchangeclusterattributes.Syntax:haclusoptionargOptions:-display-help-modify-modifymodify_options-valueattribute-notesExample:haclusvalueClusterLocation,Page40,StartupStatesandTransitions启动的状态和迁移,Page41,ShutdownStatesandTransitions停止的状态和迁移,RUNNING,LEAVING,EXITING,EXITED,EXITING_FORCIBLY,FAULTED,hastop,hastop-force,Resourcesofflined,agentsstopped,Unexpectedexit,Page42,VCSTroubleshooting,从以下几个方面来监控VCS：,VCS的日志文件系统的日志文件使用hastatus命令查看VCS的状态SNMP事件告警机制集群管理图形界面clustermanager,Page44,VCSLogEntries,VCS引擎日志:/var/VRTSvcs/log/engine_A.log通过GUI图形界面查看日志或者hamsg命令:hamsgengine_AExampleentries:TAG_D2001/04/0312:17:44VCS:11022:VCSengine(had)startedTAG_D2001/04/0312:17:44VCS:10114:openingGABlibraryTAG_C2001/04/0312:17:45VCS:10526:IpmHandle:recvpeerexitederrno10054TAG_E2001/04/0312:17:52VCS:10077:receivednewclustermembershipTAG_E2001/04/0312:17:52VCS:10080:Membership:0 x3,Jeopardy:0 x0,Page45,代理日志：AgentLogEntries,代理日志在/var/VRTSvcs/log目录下面日志文件用AgentName_A.log来命名，如：IP_A.log日志级别的设置:noneerror(默认设置)infodebugall通过命令来改变日志级别:hatype-modifyres_typeLogLeveldebug,Page46,集群通信问题解决：,使用命令hastatussummary检查VCS如果输出类似如下，则表明集群之间的通信有问题VCS:11307:Nodehasnotreceivedclustermembershipyet,cannotprocessHAcommand如果输出类似如下，则表明VCS的引擎启动有问题hatest1STALEADMINWAIT:allsystemstale首先用lltconfig命令检查llt模块是否是running状态，如果不是检查/etc/llttab文件,Page47,LLT模块问题解决：,检查/etc/llthost文件，主机名必须与/etc/llttab中的主机名保持一致，主机序列号必须在0-31范围内如果llt的状态是running，用命令lltstatn检查是否所有的心跳线都是好的，请先确认在/etc/llttab中配置的网卡是否都是UP状态的，可以用ifconfig查看，类似输出如下：LLTnodeinformation:NodeStateLinks*0test-smc3OPEN31storage-1OPEN3,Page48,GAB模块问题解决：,首先检查GAB模块是否已经运行，gabconfiga如果输出如下，则表明GAB模块有问题，请检查/etc/gabtab文件，GABPortMemberships如果GAB一起动马上关闭了，请检查LLT模块是否有问题如果没有h端口的输出则表明HAD有问题，正常的输出如下：GABPortMemberships=Portagena76401membership01Porthgena76404membership01,Page49,HAD模块问题解决,首先确认LLT模块和GAB模块已经正确启动使用hacfverify/etc/VRTSvcs/conf/config检查VCS的配置文件是否配置正确，无输出则表明是正确的确认VCS的license是否是正确的：vxlicrep，如果输出类似如下，则需要重新输入licensevxlicrepERRORV-21-3-1003TherearenovalidVERITASLicensekeysinstalledinthesystem.重新输入有效的license，使用命令vxlicinst，按照提示输入license使用命令hastatus-sum查看状态STALE_ADMIN_WAIT:ThesystemhasastaleconfigurationandnoothersystemisinaRUNNINGstate.ADMIN_WAIT:Thesystemcannotbuildorobtainavalidconfiguration.,Page50,STALE_ADMIN_WAIT,TorecoverfromSTALE_ADMIN_WAITstate:从这个状态恢复Visuallyinspectthemain.cffiletodeterminewhetheritisvalid.验证配置文件是否正确Editthemain.cffile,ifnecessary.如有必要修改该文件Verifythesyntaxofmain.cf,ifmodified.修改之后验证语法的正确性hacfverifyconfig_dirStartVCSonthesystemwiththevalidmain.cffile:强制启动VCS使用有效的配置文件hasys-forcesystem_nameAllothersystemsperformaremotebuildfromthesystemnowrunning.其他的节点可以通过这个启动的节点进行远程启动,Page51,ADMIN_WAIT,AsystemcanbeintheADMIN_WAITstateunderthesecircumstances:下列情形之一可能会出现这个状态A.staleflagexistsandthemain.cffilehasasyntaxproblem.配置文件有问题Adiskerroroccursaffectingmain.cfduringalocalbuild.本地启动的时候硬盘有问题Thesystemisperformingaremotebuildandlastrunningsystemfails.该节点正在远程启动，结果那个节点失效了Restoremain.cfandusetheprocedureforSTALE_ADMIN_WAIT.,Page52,IdentifyingOtherProblems其他问题的确定,AfterverifyingthatHAD,LLT,andGABarefunctioningproperly,runhastatussumtoidentifyproblemsinotherareas:在检查了HAD，LLT和GAB正确之后就要使用hastatussum来确定其他区域的问题Servicegroups服务组Resources资源Agentsandresourcetypes代理和资源类型,Page53,ServiceGroupProblems:GroupNotConfiguredtoStartorRun服务组的问题，没有配置为自动启动,ServicegroupnotonlinedautomaticallywhenVCSstarts:CheckAutoStartandAutoStartListattributes:VCS启动的时候服务没有自动online，先检查AutoStart和AutoStartList这两个属性hagrpdisplayservice_groupServicegroupnotconfiguredtorunonthesystem:服务组没有配置为在这个节点上运行ChecktheSystemListattribute.检查SystemList属性Verifythatthesystemnameisincluded.确认这个节点属于这个集群,Page54,ServiceGroupAutoDisabled服务组自动失效,Autodisableoccurswhen:由下列情形会发生自动失效GABseesasystembuthadisnotrunningonthesystem.节点已经运行gab，但是没有启动VCS的hadResourcesoftheservicegrouparenotfullyprobedonallsystemsintheSystemList.在所有的检点上服务组的资源没有全部探测到Aparticularsystemisvisiblethroughdiskheartbeatonly.通过磁盘心跳只有部分节点是可见的MakesurethattheservicegroupisofflineonallsystemsinSystemListattribute.确认这个服务组在所有的节点上都是offline的CleartheAutoDisabledattribute:清除自动失效属性hagrpautoenableservice_group-syssystemBringtheservicegrouponline.将这个服务组online,Page55,ServiceGroupNotFullyProbed服务组没有全部探测到,Usuallyaresultofimproperlyconfiguredresourceattributes:通常是资源的属性没有正确的配置CheckProbesPendingattribute:检查这个属性hagrp-displayservice_groupCheckwhichresourcesarenotprobed:查看哪个资源没有探测到hastatus-sumCheckProbesattributeforresources:检查资源的属性hares-displayToproberesources:探测这个资源haresproberesource-syssystem,Page56,ServiceGroupFrozen服务组冻结,VerifyvalueofFrozenandTFrozenattributes:确认这两个属性的值hagrp-displayservice_groupUnfreezetheservicegroup:解冻这个服务组hagrp-unfreezegroup-persistentIfyoufreezepersistently,youmustunfreezepersistently.如果是持久冻结，解冻的时候必须要是持久解冻,Page57,ServiceGroupIsNotOfflineElsewhere服务组在任何地方都没有offline,Determinewhichresourcesareonline/offline:确定哪些资源是online和offline的hastatus-sumVerifytheStateattribute:确认状态属性hagrp-displayservice_groupOfflinethegroupontheothersystem:在其他节点offline这个服务组hagrp-offlineFlushtheservicegroup:使这个服务组可以被部分拉起hagrp-flushservice_group-syssystem,Page58,ServiceGroupWaitingforResource服务组在等待某个资源,ReviewIstateattributeofallresourcestodeterminewhichresourceiswaitingtogoonline.查看哪个资源正在等待online的过程中Usehastatustoidentifytheresource.使用hastauts来确认这个资源Makesuretheresourceisoffline(attheoperatingsystemlevel).Cleartheinternalstateoftheservicegroup:hagrpflushservice_group-syssystemBringallotherresourcesintheservicegroupofflineandtrytobringtheseresourcesonlineonanothersystem.VerifythattheresourceworksproperlyoutsideVCS.Checkforerrorsinattributevalues.,Page59,IncorrectLocalName主机名不一致,Aservicegroupcannotbebroughtonlineifthesystemnameisinconsistentinllthosts,llttab,ormain.cffiles.如果在llthosts，llttab和main.cf中的主机名不一致则这个服务组不会被onlineCheckeachfileforconsistentuseofsystemnames.检查这些文件Correctanydiscrepancies.修改成一致的Ifmain.cfischanged,stopandrestartVCS.如果main.cf被修改了，停止和重启VCSIfltthostsorltttabischanged:如果llthosts和llttab修改了，停止VCS，gab，和llt，重新启动llt，gab和VCSStopVCS,GAB,andLLT.RestartLLT,GAB,andVCS.,Page60,ConcurrencyViolations网络冲突,Occurswhenafailoverservicegroupisonlineorpartiallyonlineonmorethanonesystem失效转移类型的服务组在多个节点上运行就会导致冲突NotificationprovidedbytheViolationtrigger:InvokedonthesystemthatcausedtheconcurrencyviolationNotifiestheadministratorandtakestheservicegroupofflineonthesystemcausingtheviolationConfiguredbydefaultwiththeviolationscriptin/opt/VRTSvcs/bin/triggersCanbecustomized:Sendmessagetothesystemlog.Displaywarningonallclustersystems.Sende-mailmessages.,Page61,ServiceGroupWaitingforResourcetoGoOffline服务组等待资源offline,Identifywhichresourceisnotoffline:确定哪个资源没有offlinehastatussummaryChecklogs.检查日志Manuallybringtheresourceoffline,ifnecessary.必要的时候手动offline这个资源ConfigureResNotOfftriggerfornotificationoraction.可以配置ResNotOfftrigger这个处罚脚本，一旦发生这种情况可以报告给管理员,Page62,ResourceProblems:UnabletoBringResourcesOnline资源问题：不能将某个资源online,Possiblecausesoffailurewhilebringingresourcesonline:不能将资源online的原因Waitingforchildresources等待子资源StuckinaWAITstate在一个等待状态Agentnotrunning代理没有运行,Page63,ProblemsBringingResourcesOffline资源offline的问题,Waitingforparentresourcestocomeoffline等待父资源offlineWaitingforaresourcetorespond等待这个资源的响应Agentnotrunning代理没有运行,Page64,CriticalResourceFaults严重资源错误,Determinewhichcriticalresourcehasfaulted:查看严重资源错误hastatussummaryMakesurethattheresourceisoffline.确认这个资源已经offlineExaminetheenginelog.检查日志Fixtheproblem.修复问题VerifythattheresourcesworkproperlyoutsideofVCS.确认这个资源可以在VCS之外正确运行ClearfaultinVCS.在VCS中清除fault状态,Page65,ClearingFaults清除faults,Afterexternalproblemsarefixed:在外部的错误修正后Clearanyfaultsonnonpersistentresources.清除非持久的资源的错误hares-clearresource-syssystemCheckattributefieldsforincorrectormissingdata.检查不正确的配置属性Ifservicegroupispartiallyonline:Flushwaitstates:hagrp-flushservice_group-syssystemBringresourcesofflinefirstbeforebringingthemonline.,Page66,AgentProblems:AgentNotRunning代理的问题：代理没有运行,DeterminewhethertheagentforthatresourceisFAULTED:确认那个代理的资源是否使FAULTED状态的hastatussummaryUsethepscommandtoverifythattheagentprocessisnotrunning.使用ps命令确认代理的进程没有运行Chec