linux(Red Hat)系统故障的记录器--日志文件.doc

Linux系统故障的记录器日志文件当系统发生异常时，除直接宕机外，一般是有告警消息的。我们可依据网管界面显示的告警消息，使用相关命令对相关对象进行检查。当告警消息不足以分析故障原因时，就需要分析服务器上的系统日志文件了。Linux在默认安装的情况下，记录系统运行消息的日志文件主要都存放在/var/log目录下，并自动进行新老轮替（由/etc/logrotate.conf配置文件决定）。其中消息涵盖范围最广的是messages文件，其他的有：dmesg，boot，cron，secure，utmp，wtmp，btmp等文件。这些文件的内容完全取决于/etc/syslog.conf配置文件中的配置，即该配置文件决定了任何一条消息（消息类型，消息优先级，消息目的地）存放到何处。另外，服务器上的各业务应用程序都有各自的运行日志文件，其存放位置和解读方法由各业务应用程序决定，请见支持厂商的维护手册。首先看一下/var/log目录下有哪些日志文件（默认安装）：rootlocalhost jc_log# cd /var/logrootlocalhost log# ls -ltotal 4424-rw-r- 1 root root 780 Feb 26 15:18 acpiddrwx- 2 root root 4096 Apr 11 2008 aidedrwx-S- 2 amanda disk 4096 Jun 29 2004 amanda-rw- 1 root root 15496 May 12 2009 anaconda.log-rw- 1 root root 22968 May 12 2009 anaconda.syslog-rw- 1 root root 43582 May 12 2009 anaconda.xlogdrwxr-x- 2 root root 4096 Apr 11 2008 audit-rw- 1 root root 0 Apr 4 04:02 boot.log-rw- 1 root root 240 Apr 4 04:02 boot.log.1-rw- 1 root root 240 Mar 28 04:02 boot.log.2-rw- 1 root root 240 Mar 21 04:02 boot.log.3-rw- 1 root root 240 Mar 14 04:02 boot.log.4drwxr-xr-x 2 canna canna 4096 Nov 1 2004 canna-rw- 1 root root 338025 Apr 9 19:05 cron-rw- 1 root root 420911 Apr 4 04:02 cron.1-rw- 1 root root 420362 Mar 28 04:02 cron.2-rw- 1 root root 420762 Mar 21 04:02 cron.3-rw- 1 root root 420744 Mar 14 04:02 cron.4drwxr-xr-x 2 lp sys 4096 Apr 4 04:02 cups-rw-r-r- 1 root root 21400 Feb 26 15:17 dmesgdrwxr-x- 2 exim exim 4096 Aug 25 2005 eximdrwxr-xr-x 2 root root 4096 Feb 26 15:21 gdmdrwx- 2 root root 4096 May 9 2008 httpddrwxr-xr-x 2 htt htt 4096 Nov 13 2007 iiimdrwx- 2 root root 4096 May 12 2009 iptraf-r- 1 root root 146584 Apr 9 15:46 lastlogdrwxr-xr-x 2 root root 4096 May 12 2009 mail-rw- 1 root root 6402 Apr 9 04:02 maillog-rw- 1 root root 8964 Apr 4 04:02 maillog.1-rw- 1 root root 8980 Mar 28 04:02 maillog.2-rw- 1 root root 8966 Mar 21 04:02 maillog.3-rw- 1 root root 8970 Mar 14 04:02 maillog.4drwxrwsr-x 2 root mailman 4096 Jul 31 2007 mailman-rw- 1 root root 1055 Apr 9 19:09 messages-rw- 1 root root 1574 Apr 4 04:02 messages.1-rw- 1 root root 1399 Mar 28 04:02 messages.2-rw- 1 root root 1667 Mar 21 04:02 messages.3-rw- 1 root root 2574 Mar 14 04:02 messages.4-rw-r- 1 mysql mysql 235 May 14 2009 mysqld.logdrwxr-xr-x 3 news news 4096 May 12 2009 news-rwx- 1 postgres postgres 0 May 12 2009 pgsqldrwx- 2 root root 4096 Nov 2 2004 ppp-rw-r-r- 1 root root 56217 Apr 8 04:02 prelink.logdrwxrwx- 2 quagga quagga 4096 Feb 5 2008 quaggadrwx- 3 radiusd radiusd 4096 Apr 1 04:02 radius-rw-r-r- 1 root root 51445 Apr 9 04:02 rpmpkgs-rw-r-r- 1 root root 51445 Apr 3 04:02 rpmpkgs.1-rw-r-r- 1 root root 51445 Mar 27 04:02 rpmpkgs.2-rw-r-r- 1 root root 51445 Mar 20 04:02 rpmpkgs.3-rw-r-r- 1 root root 51445 Mar 13 04:02 rpmpkgs.4drwxr-xr-x 2 root root 4096 Apr 9 00:00 sadrwx- 2 root root 4096 Jun 20 2008 samba-rw-r-r- 1 root root 75485 May 12 2009 scrollkeeper.log-rw- 1 root root 226898 Apr 9 18:59 secure-rw- 1 root root 287738 Apr 4 03:59 secure.1-rw- 1 root root 287019 Mar 28 03:59 secure.2-rw- 1 root root 287661 Mar 21 04:00 secure.3-rw- 1 root root 291842 Mar 14 03:59 secure.4-rw-r-r- 1 root root 65 Apr 4 04:03 snmpd.log-rw-r-r- 1 root root 115 Apr 4 04:02 snmpd.log.1-rw-r-r- 1 root root 115 Mar 28 04:02 snmpd.log.2-rw-r-r- 1 root root 115 Mar 21 04:02 snmpd.log.3-rw-r-r- 1 root root 115 Mar 14 04:02 snmpd.log.4-rw- 1 root root 0 Apr 4 04:02 spooler-rw- 1 root root 0 Mar 28 04:02 spooler.1-rw- 1 root root 0 Mar 21 04:02 spooler.2-rw- 1 root root 0 Mar 14 04:02 spooler.3-rw- 1 root root 0 Mar 7 04:02 spooler.4drwxr-x- 2 squid squid 4096 Apr 1 2008 squiddrwxr-xr-x 2 uucp uucp 4096 May 12 2009 uucpdrwxr-xr-x 2 root root 4096 Feb 7 2008 vboxdrwxr-xr-x 2 root root 4096 Apr 5 03:21 VRTSpbx-rw-rw-r- 1 root utmp 4992 Apr 9 17:57 wtmp-rw-rw-r- 1 root utmp 28032 Mar 31 13:50 wtmp.1-rw- 1 root root 0 Apr 4 04:02 xferlog-rw- 1 root root 0 Mar 28 04:02 xferlog.1-rw- 1 root root 0 Mar 21 04:02 xferlog.2-rw- 1 root root 0 Mar 14 04:02 xferlog.3-rw- 1 root root 0 Mar 7 04:02 xferlog.4-rw-r-r- 1 root root 51579 Feb 26 15:22 Xorg.0.log-rw-r-r- 1 root root 51562 Feb 12 10:04 Xorg.0.log.oldrootlocalhost log#注：其中有5个同名文件的轮替周期为星期，例messages；有2个同名文件的轮替周期为月，例wtmp。1 /var/log/messages文件（文本文件） 用于分析系统故障原因。 该文件记录了几乎所有系统运行的消息，所以这是分析系统故障的最主要的日志文件。这与AIX操作系统有故障会给出errpt是完全不同的。这应该是开源软件的特点没有故障库，只管源端（各进程）发消息，目的端（各日志文件）收消息，而消息解读完全由使用者自已完成，这也是linux系统维护的难点之一。该文件中的消息格式为：日期及时间，主机名，程序名（包含PID或内核标识的方括号），冒号和一个空格，最后是消息内容。（1） 日期及时间：由于整个智能网的NTP时间服务已经部署，利用该“日期 及时间”可快速定位故障源，因为在一个较短的时间段内可能有多个设备先后报警。（2） 主机名，程序名：组合成消息发源地（3） 消息主体：具体描述了告警内容，这是分析判断故障原因的主要依据。但由于messages文件记录了几乎所有系统运行的消息，范围很大（软件，硬件；系统，服务，网络，应用），又没有相关的告警消息参考资料，故上网查询是一个行之有效的方法。 可使用more,cat,head,tail,grep等命令来直接查看messages文件。rootlocalhost log# cat messages.1Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232195Feb 13 03:41:20 localhost kernel: Aborting journal on device sdc1.Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232196Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232197Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232198Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232199Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232200Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232201Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232202Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232203Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232204Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232205Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_free_blocks_sb: bit already cleared for block 244232206Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_reserve_inode_write: Journal has abortedFeb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_truncate: Journal has abortedFeb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_reserve_inode_write: Journal has abortedFeb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_orphan_del: Journal has abortedFeb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_reserve_inode_write: Journal has abortedFeb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1) in ext3_delete_inode: Journal has abortedFeb 13 03:41:20 localhost kernel: ext3_abort called.Feb 13 03:41:20 localhost kernel: EXT3-fs error (device sdc1): ext3_journal_start_sb: Detected aborted journalFeb 13 03:41:20 localhost kernel: Remounting filesystem read-only案例分析：在Feb 13 03:41:20，设备名为sdc1的文件系统开始报错（第1条），其后所有消息表示：该ext3文件系统的日志功能丧失，该文件系统已经无法仍为ext3。为保证sdc1中的文件数据的一致性，系统将此文件系统的工作模式从“读写”置为“只读”（最后一