H852小型机故障记录

小型机故障记录故障机器：主机名 H852故障时间：2007年9月3日下午2点左右故障现象：同一网段的小型机与该主机通信正常，但跨网段的机器无法与该主机通信初步判断：该主机的网络设置被改动，或者是网络设备物理或逻辑上的变动所致处理过程：1询问网络管理员，他们最近没有对网络设备作物理或逻辑上的调整2在本机查网络设置和日志通过对本机的配置进行查询，发现存在如下现象，现详细记录如下：#hostnameH852#ifconfig-aenO:flags=4e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>enl:flags=4e080863,80<UP,BROADCAST,NOTRAILERS,RUNNINQSIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>loO:flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNINQSIMPLEX,MULTICAST,GROUPRT,64BIT>inetnetmaskOxffOOOOOObroadcast55inet6::l/0tcp_sendspace131072tcp_recvspace131072rfcl32313#netstat-rRoutingtablesDestinationGateway FlagsRefsUseIfPMTUExpGroupsRouteTreeforProtocolFamily2(Internet):defaultUG01312547enO - -UHSb00enl - =>10/8U311028enl - -loopbackUGHS02loO - -loopbackUHSb00loO - -二>h852_serviceUHSb00enO - =>h852_serviceU5104741enO - -h852_serviceloopbackUGHS016668loO - - -loopbackUHSb0269loO - - -二>h852_serviceUHSb02enO - -UHSb00enl - -rootpts/1Apr2010:58-10:59(00:00)rootpts/1Apr2010:48-10:48(00:00)rootpts/1Apr2010:38-10:38(00:00)rootpts/1Apr2010:28-10:28(00:00)rootpts/1Apr2010:18-10:18(00:00)rootpts/1Apr2010:08-10:08(00:00)rootpts/1Apr2009:58-09:58(00:00)rootpts/1Apr2009:47-09:47(00:00)rootpts/1Apr2009:37-09:37(00:00)rootpts/1Apr2009:27-09:27(00:00)rootpts/1Apr2009:17-09:17(00:00)rootpts/1Apr2009:07-09:07(00:00)rootpts/1Apr2008:57-08:57(00:00)rootpts/1Apr2008:46-08:46(00:00)rootpts/1Apr2008:36-08:36(00:00)rootpts/1Apr2008:26-08:26(00:00)rootpts/1Apr2008:16-08:16(00:00)rootpts/1Apr2008:06-08:06(00:00)rootpts/1Apr2007:56-07:56(00:00)rootpts/1Apr2007:45-07:45(00:00)rootpts/1Apr2007:35-07:35(00:00)rootpts/1Apr2007:25-07:25(00:00)通过对以上信息的分析，问题在于网卡enl的IP被人为改动所致在以上出错信息中，9月3日的报错是因为重启HACMP的缘故，不是事故的原因。查以上信息，enl的IP应该在2007年7月16日以前被改动，具体改成什么IP,不可查知，但在9月3日出现故障时，可查知enl的IP被改为（正常IP应是0）故障原因分析：因为enl的IP被改为与主网卡（enO）同一个A网段的地址，所以，网络会发生聚合一-就是该主机会把诸如或之类所有的地址全部当成是网段的主机。所以，当其它网段（例如IP为10.20.222..29的终端）的TCP请求从到达主机后，因为网络地址聚合的原理，该主机会认为从10.20.222.・29发来的请求包是来自本地网，所以回包信息不会发给它的网关,所以，到不同网段的TCP包也就不会被路由出去，而是在本局域网内传递（这就是为什么跨网段的终端不能访问到该主机，而同一局域网的小型机能访问该主机的原因）最后，试图用smittcpip改回enl的IP时，发现其IP设置是正确的（为0）,因此推测出对enl的IP的改动是通过ifconfigenlnetmaskup命令动态实现的。直接用ifconfig将IP改回0,网络恢复正常。127/8 loopback U2114059043IoORouteTreeforProtocolFamily24(Internetv6):::1 ::1 UH0 0loO#4#en-pt#errptIDENTIFIERTIMESTAMPTCRESOURCE_NAMEDESCRIPTIONAFA8990592E96A0D97419D605294E5FF6D19271E28854E8199FA80C79D30B78E9D30B78E9D30B78E9D30B78E9D30B78E9D30B78E9D30B78E3C81E43F92E96A0D0903163807IOgrpsvcs0903163807PUtopsvcs0903163807IOtopsvcs0903163807IOtopsvcs0903143807IOtopsvcs0903143807IOgrpsvcs0903143807UShaemd0826234007TSttyl0825124307TStty10825050007TSttyl0822205307TSttyl0821200307TSttyl0818012907TSttyl0812174207TSttyl0720000007PUtopsvcs0716171507PUtopsvcsGroupServicesdaemonstartedLocaladaptermisconfigurationdetectedTopologyServicesdaemonstartedMissingorincorrectadapteraddressTopologyServicesdaemonstoppedGroupServicesdaemonstoppedSOFTWARERECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTLateinsendingheartbeatLocaladaptermisconfigurationdetected5#errpt-aj5294E5FFLABEL:IDENTIFIER:TS_FDJNVAL_ADDR_ST5294E5FFDate/Time:MonSep316:38:36BEIST2007SequenceNumber:95368MachineId:000C4ADF4C00NodeId:h852Class:OType:ResourceName:INFOtopsvcsDescriptionMissingorincorrectadapteraddressProbableCausesAdapterisnotconfiguredAdapterhasnoaddressAdapterhasanaddressthatisoutsidetheconfigurationUserCausesAdapterwasincorrectlyconfiguredRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesFailureCausesAproblemoccurredwhenconfiguringtheadapterRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesDetailDataDETECTINGMODULErsct,force_down.C,1.11,501ERRORID6zLCZG/AUwq4/w8P.SEV08 REFERENCECODEInterfacenameenlActualaddressExpectedboot-timeaddress6#errpt-aj92E96A0DLABEL: TS_MISCFG_EMIDENTIFIER: 92E96A0DDate/Time: MonSep316:38:37BEIST2007SequenceNumber:95370MachineId:000C4ADF4C00NodeId:h852Class:UType:PENDResourceName:topsvesResourceClass:NONEResourceType:NONELocation:VPD:DescriptionLocaladaptermisconfigurationdetectedProbableCausesLocaladapterassignedincorrectIPaddressInstallCausesLocaladapterassignedincorrectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0BUwq4/2DC/SEV08 REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddressLABEL:TS_MISCFG_EMIDENTIFIER:92E96A0DDate/Time:Date/Time:MonJul1617:15:35BEIST2007SequenceNumber:95344MachineId:000C4ADF4C00h852h852NodeId:Class:Type:ResourceName:PENDResourceClass:topsvcsNONEResourceType:Location:NONEVPD:DescriptionLocaladaptermisconfigurationdetectedProbableCausesLocaladapterassignedincorrectIPaddressInstallCausesLocaladapterassignedincon-ectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0rQna4/yPD1SEV08 REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddress7部分登录信息#lastrootpts/2RAC_P5711Sep0409:31-09:44(00:13)rootpts/1RAC_P5711Sep0409:17stillloggedin.rootpts/3h85I_serviceSep0316:51-17:04(00:13)rootpts/2RAC_P5711S叩0316:47-16:58(00:11)rootpts/1h851_serviceSep0316:42-18:57(02:15)rootIftOSep0313:55-16:40(02:44)rootpts/1Aug2415:24-16:53(01:29)rootpts/1Aug2410:14-12:24(02:10)rootpts/1Aug2120:05-20:11(00:05)rootpts/1Aug2119:30-19:31(00:01)rootpts/1h851_serviceAug2115:20-16:57(01:37)rootIftOAug1416:27-16:55(00:27)rootdtremote58:0Aug1409:03-13:38(04:35)rootpts/1Aug0911:08-15:41(04:33)rootdtremote58:0Aug0910:34-10:43(00:09)rootpts/OAug0308:59-13:16(04:17)rootpts/1h851_serviceJul1816:34-16:36(00:01)rootpts/1Jul1716:47-23:18(06:31)rootpts/1Jul1708:41-14:18(05:37)rootpts/1Jul1618:00-18:02(00:02)rootdtremote1:1Jul1616:58-17:19(00:20)rootpts/1Jul1616:48-16:55(00:07)rootrootIftOpts/OJul1616:28-17:03(11+00:34)Jul1616:25-16:26(00:00)rootpts/OMay2209:28-09:29(00:00)rootpts/OMay1615:43-15:43(00:00)rootpts/OApr2623:26-23:28(00:02)rootpts/OApr2610:04-10:08(00:04)rootpts/OApr2509:51-11:31(01:39)rootrebootshutdownrootrootIftOpts/OIftOpts/Oh851_stbyApr0610:14-16:19(1014-06:05)Apr0610:09Apr0610:00Apr0609:45-10:00(00:14)Aug2511:05-11:05(00:00)rootpts/Oh851_stbyAug2115:07-15:24(00:17)rootpts/Oh851_stbyAug2110:53-11:00(00:06)rootrebootshutdownrootrebootrebootshutdownrootrebootshutdownoracleIftOIftOIftOIftOIftOIftOftpJul2519:41-10:15(203+14:33)Jul2519:37Jul2514:56Jun2211:42-14:55(33+03:13)Jun2112:29Apr2810:54Apr2810:28Apr2810:27-10:28(00:01)Apr2810:23Apr2809:56Apr2708:58-08:58(00:00)nlastn1432lines,108241charactersshutdownIftOApr2809:56Apr2708:58-08:58(00:00)rootpts/2Apr2610:48-10:53(00:05)oracleftpApr2609:45-09:45(00:00)oracleftpApr2609:32-09:32(00:00)oracleftpApr2608:58-08:58(00:00)oracleftpApr2509:01-09:02(00:01)oracle即Apr2408:59-09:00(00:01)oraclepts/1Apr2115:36-15:48(00:11)rootpts/1Apr2110:01-10:01(00:00)rootpts/1Apr2109:51-09:51(00:00)rootpts/1Apr2109:40-09:41(00:00)rootpts/1Apr2109:30-09:30(00:00)rootpts/1Apr2109:20-09:20(00:00)rootpts/1Apr2109:10-09:10(00:00)rootpts/1Apr2109:00-09:00(00:00)rootpts/1Apr2108:50-08:50(00:00)rootpts/1Apr2108:39-08:40(00:00)rootpts/1Apr2108:29-08:29(00:00)rootpts/1Apr2108:19-08:19(00:00)rootpts/110.0.1,199Apr2108:09-08:09(00:00)rootpts/1Apr2107:59-07:59(00:00)rootpts/1Apr2107:49-07:49(00:00)rootpts/1Apr2107:38-07:39(00:00)rootpts/1Apr2107:28-07:28(00:00)rootpts/1Apr2107:18-07:18(00:00)rootpts/1Apr2107:08-07:08(00:00)rootpts/1Apr2106:58-06:58(00:00)rootpts/1Apr2106:48-06:48(00:00)rootpts/1Apr2106:37-06:38(00:00)rootpts/1Apr2106:27-06:27(00:00)rootpts/1Apr2106:17-06:17(00:00)rootpts/1Apr2106:07-06:07(00:00)rootpts/1Apr2105:57-05:57(00:00)rootpts/1Apr2105:47-05:47(00:00)rootpts/1Apr2105:37-05:37(00:00)rootpts/1Apr2105:26-05:26(00:00)rootpts/1Apr2105:16-05:16(00:00)rootpts/1Apr2105:06-05:06(00:00)rootpts/1Apr2104:56-04:56(00:00)rootpts/1Apr2104:46-04:46(00:00)rootpts/1Apr2104:36-04:36(00:00)rootpts/1Apr2104:25-04:25(00:00)rootpts/1Apr2104:15-04:15(00:00)rootpts/1Apr2104:05-04:05(00:00)rootpts/1Apr2103:55-03:55(00:00)rootpts/1Apr2103:45-03:45(00:00)rootpts/1Apr2103:55-03:55(00:00)rootpts/1Apr2103:45-03:45(00:00)rootpts/1Apr2103:35-03:35(00:00)rootpts/1Apr2103:24-03:24(00:00)rootpts/1Apr2103:14-03:14(00:00)rootpts/1Apr2103:04-03:04(00:00)rootpts/1Apr2102:54-02:54(00:00)rootpts/110.0.1,199Apr2102:44-02:44(00:00)rootpts/1Apr2102:34-02:34(00:00)rootpts/1Apr2102:23-02:23(00:00)rootpts/1Apr2102:13-02:13(00:00)rootpts/1Apr2102:03-02:03(00:00)rootpts/1Apr2101:53-01:53(00:00)rootpts/1Apr2101:43-01:43(00:00)rootpts/1Apr2101:33-01:33(00:00)rootpts/1Apr2101:22-01:22(00:00)rootpts/1Apr2101:12-01:12(00:00)rootpts/1Apr2101:02-01:02(00:00)rootpts/1Apr2100:52-00:52(00:00)rootpts/110.0.1,199Apr2100:42-00:42(00:00)rootpts/1Apr2100:32-00:32(00:00)rootpts/1Apr2100:21-00:22(00:00)rootpts/1Apr2100:11-00:11(00:00)rootpts/1Apr2100:01-00:01(00:00)rootpts/1Apr2023:51-23:51(00:00)rootpts/1Apr2023:41-23:41(00:00)rootpts/1Apr2023:31-23:31(00:00)rootpts/1Apr2023:20-23:20(00:00)rootpts/1Apr2023:10-23:10(00:00)rootpts/1Apr2023:00-23:00(00:00)rootpts/1Apr2022:50-22:50(00:00)rootpts/1Apr2022:40-22:40(00:00)rootpts/1Apr2022:30-22:30(00:00)rootpts/1Apr2022:19-22:20(00:00)rootpts/1Apr2022:09-22:09(00:00)rootpts/1Apr2021:59-21:59(00:00)rootpts/1Apr2021:49-21:49(00:00)rootpts/1Apr2021:39-21:39(00:00)rootpts/1Apr2021:29-21:29(00:00)rootpts/1Apr2021:18-21:19(00:00)rootpts/1Apr2021:08-21:08(00:00)rootpts/1Apr2020:58-20:58(00:00)rootpts/1Apr2020:48-20:48(00:00)rootpts/1Apr2020:38-20:38(00:00)rootpts/1Apr2020:28-20:28(00:00)rootpts/1Apr2020:17-20:18(00:00)rootpts/1Apr2020:07-20:07(00:00)rootpts/1Apr2020:17-20:18(00:00)rootpts/1Apr2020:07-20:07(00:00)rootpts/1Apr2019:57-19:57(00:00)rootpts/1Apr2019:47-19:47(00:00)rootpts/1Apr2019:37-19:37(00:00)rootpts/1Apr2019:27-19:27(00:00)rootpts/110.0.1,199Apr2019:16-19:17(00:00)rootpts/1Apr2019:06-19:06(00:00)rootpts/1Apr2018:56-18:56(00:00)rootpts/1Apr2018:46-18:46(00:00)rootpts/1Apr2018:36-18:36(00:00)rootpts/1Apr2018:26-18:26(00:00)rootpts/1Apr2018:15-18:16(00:00)rootpts/1Apr2018:05-18:05(00:00)rootpts/1Apr2017:55-17:55(00:00)rootpts/1Apr2017:45-17:45(00:00)rootpts/1Apr2017:35-17:35(00:00)rootpts/1Apr2017:25-17:25(00:00)rootpts/110.0.1,199Apr2017:15-17:15(00:00)rootpts/1Apr2017:04-17:04(00:00)rootpts/1Apr2016:54-16:54(00:00)rootpts/1Apr2016:44-16:44(00:00)rootpts/1Ap