爱立信CUDB日常维护指令

1.1 登陆操作 CUDB节点中有三类板卡，分别是GEP3板，SCXB(DMX)板和 NWI-E板。 我们需要登录这些板子收集相应的日志，可以用SecureCRT，terminal或者其他SSH客户软件登录这些板卡。有两种方式可以登陆到CUDB：1） Console直连Console直连的方式在日常操作维护中不推荐使用。通过Console直连的操作一般为对于硬件的操作，如更换板卡。CUDB系统Console连接配置表。硬件名称波特率数据位奇偶校验停止位流控SCXB8None1NoneGEP38None1NoneNWI-E96008None1None2） 通过网管网络连接在对于CUDB的日常操作维护时，推荐通过网管网络连接CUDB。从OSS登陆SC板卡和DMX板卡使用SSH协议，登陆NWI使用TELNET协议。CUDB系统网管登陆信息表登陆节点登陆方式端口用户名密码登陆命令CUDB GEP3SSH22rootrootrootssh root DMXSSH2024expertexpertssh expert -p 2024NWITelnet23admintelnet 1.2 CUDB 系统检查通常情况下以下检查应该包括在每日健康检查中。1.2.1 CUDB总体系统检查验证整个系统状态。在CUDB 某块SC板卡上执行这些指令。执行指令:# cudbSystemStatus命令描述:这条命令自动执行下面的系统状态检查。预期结果:Execution date: Tue Mar 25 11:29:36 CST 2014CUDB Software Version:!- CUDB DESIGN DISTRIBUTION: CUDB13B CXP/6 R1KChecking BC clusters:Site 1 SM leader: Node 1 OAM2 Node 10.173.0.2 BC server in SC_2_1 . running BC server in SC_2_2 . running (Leader) BC server in PL_2_5 . runningSite 2 NoLeader Node 10.173.0.34 BC server in SC_2_1 . running BC server in SC_2_2 . running BC server in PL_2_5 . runningChecking System Monitor BC status in local node: SM-BC in OAM1 . running SM-BC in OAM2 . runningChecking Clusters status:Node 1: PL Cluster (2%) .OK DSG1 Cluster (1%) .OK DSG2 Cluster (1%) .OK DSG3 Cluster (1%) .OK DSG4 Cluster (1%) .OK DSG5 Cluster (1%) .OK DSG6 Cluster (1%) .OK DSG7 Cluster (1%) .OK DSG8 Cluster (1%) .OK DSG9 Cluster (1%) .OK DSG10 Cluster (1%) .OK DSG11 Cluster (1%) .OK DSG12 Cluster (1%) .OK DSG13 Cluster (1%) .OKNode 2: PL Cluster (2%) .OK DSG1 Cluster (1%) .OK DSG2 Cluster (1%) .OK DSG3 Cluster (1%) .OK DSG4 Cluster (1%) .OK DSG5 Cluster (1%) .OK DSG6 Cluster (1%) .OK DSG7 Cluster (1%) .OK DSG8 Cluster (1%) .OK DSG9 Cluster (1%) .OK DSG10 Cluster (1%) .OK DSG11 Cluster (1%) .OK DSG12 Cluster (1%) .OK DSG13 Cluster (1%) .OKChecking NDB status: PL NDBs (6/6) .OK DS1 NDBs (2/2) .OK DS2 NDBs (2/2) .OK DS3 NDBs (2/2) .OK DS4 NDBs (2/2) .OK DS5 NDBs (2/2) .OK DS6 NDBs (2/2) .OK DS7 NDBs (2/2) .OK DS8 NDBs (2/2) .OK DS9 NDBs (2/2) .OK DS10 NDBs (2/2) .OK DS11 NDBs (2/2) .OK DS12 NDBs (2/2) .OK DS13 NDBs (2/2) .OKChecking Replication Channels in the System: Node | 1 | 2 = PLDB _|_M_|_S1_ DSG 1 _|_M_|_S1_ DSG 2 _|_M_|_S2_ DSG 3 _|_M_|_S1_ DSG 4 _|_M_|_S1_ DSG 5 _|_M_|_S2_ DSG 6 _|_M_|_S2_ DSG 7 _|_M_|_S1_ DSG 8 _|_M_|_S2_ DSG 9 _|_M_|_S1_ DSG 10 _|_M_|_S2_ DSG 11 _|_M_|_S2_ DSG 12 _|_M_|_S1_ DSG 13 _|_M_|_S2_Printing Alarms.Mar 23 12:50:05( Preventive Maintenance Logchecker has found major error(s). )Checking MySQL server connection: MySQL Master Servers connection .OK MySQL Slave Servers connection .OK MySQL Access Servers connection .OKChecking Process:OAMs. Cluster Supervisor.Running System Monitor BC.Running Reconciliation process.Running in: OAM2 Smp-client.Running Management Server Process (ndb_mgmd).Running KeepAlive process.Running ESA.Running LDAP counter.Running Log Handler process.RunningPLs. Storage Engine process (ndbd).Running LDAP FE.Running KeepAlive process.Running MySQL server process (Master).Running MySQL server process (Slave).Running MySQL server process (Access).Running CudbNotifications process.Running LDAP FE Monitor process.RunningDSs. Storage Engine process (ndbd).Running LDAP FE.Running KeepAlive process.Running MySQL server process (Master).Running MySQL server process (Slave).Running MySQL server process (Access).Running LDAP FE Monitor process.Running1.2.2 HA状态检查在CUDB Active OAM 板卡上验证所有GEP3板加入到cluster中。执行指令:#cudbHaState预期结果:LOTC cluster uptime:-Thu Mar 27 18:13:44 2014LOTC cluster state:-Node safNode=SC_2_1 joined cluster | Thu Mar 27 18:13:44 2014Node safNode=SC_2_2 joined cluster | Thu Mar 27 18:14:23 2014Node safNode=PL_2_3 joined cluster | Thu Mar 27 18:15:21 2014Node safNode=PL_2_4 joined cluster | Thu Mar 27 18:15:25 2014.AMF cluster state:-saAmfNodeAdminState.safAmfNode=SC-1,safAmfCluster=myAmfCluster: UnlockedsaAmfNodeOperState.safAmfNode=SC-1,safAmfCluster=myAmfCluster: EnabledsaAmfNodeAdminState.safAmfNode=SC-2,safAmfCluster=myAmfCluster: UnlockedsaAmfNodeOperState.safAmfNode=SC-2,safAmfCluster=myAmfCluster: EnabledsaAmfNodeAdminState.safAmfNode=PL-3,safAmfCluster=myAmfCluster: UnlockedsaAmfNodeOperState.safAmfNode=PL-3,safAmfCluster=myAmfCluster: EnabledCoreMW HA state:-CoreMW is assigned as ACTIVE in controller SC-1CoreMW is assigned as STANDBY in controller SC-2COM state:-COM is assigned as ACTIVE in controller SC-1COM is assigned as STANDBY in controller SC-2SI HA state:-saAmfSISUHAState.safSu=SC-1,safSg=2N,safApp=ERIC-CUDB_BC_SERVER_MONITOR.safSi=2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=2N,safApp=ERIC-CUDB_LDAPFE_MONITOR.safSi=2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS3_2N,safApp=ERIC-CUDB_CS.safSi=DS3_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS4_2N,safApp=ERIC-CUDB_CS.safSi=DS4_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS13_2N,safApp=ERIC-CUDB_CS.safSi=DS13_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS12_2N,safApp=ERIC-CUDB_CS.safSi=DS12_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS11_2N,safApp=ERIC-CUDB_CS.safSi=DS11_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS2_2N,safApp=ERIC-CUDB_CS.safSi=DS2_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS1_2N,safApp=ERIC-CUDB_CS.safSi=DS1_2N-1: active(1)saAmfSISUHAState.safSu=SC-1,safSg=DS7_2N,safApp=ERIC-CUDB_CS.safSi=DS7_2N-1: active(1)saAmfSISUHAState.safSu=Control1,safSg=2N,safApp=ERIC-EVIP.safSi=2N: active(1).SU States:-Status OK1.2.3 CMW状态查询在某块SC板卡上输出所有CUDB servers (OAM, PL and DS) 的磁盘使用率。执行指令:# cmw-status app csiass comp node sg si siass su pm命令描述:检查CMW状态。1.2.4 检查磁盘使用率在某块SC板卡上输出所有CUDB servers (OAM, PL and DS) 的磁盘使用率。执行指令:for a in awk /node/ print $4 /cluster/etc/cluster.conf;doecho $a; ssh $a df -h;done;命令描述:检查磁盘使用率。预期结果:SC_2_1Filesystem Size Used Avail Use% Mounted onrootfs 2.0G 1.5G 543M 74% /root 2.0G 1.5G 543M 74% /tmpfs 12G 740K 12G 1% /dev/shmshm 12G 740K 12G 1% /dev/shm/dev/sdb1 4.0G 220M 3.6G 6% /boot/dev/sdb2 9.9G 3.5G 6.0G 37% /var/log/dev/mapper/cluster_vg-data_lv 63G 11G 50G 18% /.cluster192.168.0.100:/.cluster 63G 11G 50G 18% /cluster/dev/sdb7 136G 1.2G 128G 1% /localcom_fuse_module 2.0G 1.5G 543M 74% /var/filem/nbi_rootSC_2_2Filesystem Size Used Avail Use% Mounted onrootfs 2.0G 1.5G 544M 74% /root 2.0G 1.5G 544M 74% /tmpfs 12G 740K 12G 1% /dev/shmshm 12G 740K 12G 1% /dev/shm/dev/sdb1 4.0G 220M 3.6G 6% /boot/dev/sdb2 9.9G 3.5G 5.9G 38% /var/log192.168.0.100:/.cluster 63G 11G 50G 18% /cluster/dev/sdb7 136G 1.1G 128G 1% /local1.2.5 检查网络状态输出所有CUDB servers (OAM, PL and DS) 在每个接口的网络状态。执行指令:for a in awk /node/ print $4 /cluster/etc/cluster.conf;do echo $a; ssh $a netstat -i;done;命令描述:这条命令输出系统的网络连接，路由表，接口信息，组播连接信息。用 i选项，显示所有网络接口的状态表。预期结果:CUDB1 SC_2_1 # netstat -iwarning: no inet socket available: SuccessKernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgbond0 1500 0 0 0 0 0 0 0 BMmRUbond1 1500 0 62795 0 0 0 0 0 0 BMmRUbond1:1 1500 0 - no statistics available - BMmRUbond1:2 1500 0 - no statistics available - BMmRUeth0 1500 0 0 0 0 0 0 0 BMsRUeth1 1500 0 31394 0 0 0 0 0 0 BMsRUeth2 1500 0 0 0 0 0 0 0 0 BMsRUeth3 1500 0 31401 0 0 0 0 0 0 0 BMsRUlo 16436 0 0 0 0 0 0 0 LRU1.2.6 检查CPU负载登陆某块SC板卡进行CUDB CPU负载查询。执行指令: 按 “ctrl + c” 可以退出并回到CLI模式。#cudbMpstat 命令描述:这条命令用于收集和报告每块板卡上的CPU性能统计信息。预期结果:无1.2.7 检查LDAP TPS登陆某块SC板卡进行CUDB LDAP TPS查询。执行指令: 按 “ctrl + c” 可以退出并回到CLI模式。# cudbTpsStat -d预期结果:无1.2.8 检查Active告警 在某块SC板卡执行检查哪些告警是Active的。执行指令:# fmactivealarms1.2.9 检查历史告警在两块SC板卡执行检查历史告警执行指令:# cat /var/log/ESA/alarms.log | grep -c Alarm Raise# cat /var/log/ESA/alarms.log | grep -c Alarm Clear1.2.10 检查DHCP状态登陆每块SC板卡执行指令:# /etc/init.d/dhcpd status1.2.11 检查LDAPFE进程状态登陆每块SC板卡执行指令:# for a in awk /node/ if (substr($4,1,2) = PL) print $4 /cluster/etc/cluster.conf;do ssh $a echo $a ;/etc/init.d/cudbLDAPFrontEnd status;done;1.2.12 检查ESA进程状态登陆某块SC板卡执行指令:# esaclusterstatus登陆每块SC板卡执行指令:# esa status1.2.13 检查CUDB cluster配置在某块SC板卡执行执行指令:# cat /cluster/etc/cluster.conf1.2.14 检查CUDB vipconfig配置在某块SC板卡执行执行指令：# cat /cluster/storage/system/config/*/evip.xml1.2.15 CUDB重要配置检查在主用SC板卡执行执行指令：# /opt/com/bin/cliss# show ManagedElement=1,CudbSystem=1,backboneReliability# show ManagedElement=1,CudbSystem=1,CudbLocalNode=1,CudbLdapAccess=1,ldapAttrIndexes1.2.16 CUDB数据库备份在某块SC板卡执行指令:# ls -l /home/cudb/automatedBackupStorage/*/*1.2.17 软件和配置备份在某块SC板卡执行指令:#cudbSwBackup l#cudbSwBackup p# ls -lrt /cluster/home/cudb/swbackup# ls -l /cluster/storage/no-backup# ls -l /cluster/home/cudb/oam/configMgmt/cudbSwBackup.lock1.2.18 获取CUDB counters在SC_2_1执行指令:# pmreadcounter# pmreadcounter | wc -l# ls -lrt /home/cudb/oam/performanceMgmt/output |tail -n 2001.2.19 CUDB软件在模块SC板卡执行指令:for a in awk /node/ print $4 /cluster/etc/cluster.conf;do echo $a; cmw-repository-list -node $a;done;1.2.20 LOTC crontab检查在每块SC板卡上执行命令:#crontab -l预期结果 (example): # DO NOT EDIT THIS F