aix for oracle 10g.doc

一、故障信息收集2011-12-1 logThu Dec 1 06:30:00 2011Shutting down instance: further logons disabledThu Dec 1 06:30:01 2011Stopping background process CJQ0Thu Dec 1 06:30:01 2011Stopping background process QMNCThu Dec 1 06:30:02 2011ALTER SYSTEM SET service_names= SCOPE=MEMORY SID=fmisb1;Thu Dec 1 06:30:02 2011ALTER SYSTEM SET service_names=fmisb SCOPE=MEMORY SID=fmisb1;Thu Dec 1 06:30:03 2011Stopping background process MMNLThu Dec 1 06:30:04 2011Stopping background process MMONThu Dec 1 06:30:05 2011Shutting down instance (immediate)License high water mark = 78Thu Dec 1 06:30:05 2011Stopping Job queue slave processes, flags = 7Thu Dec 1 06:30:05 2011Process OS id : 381376 alive after killErrors in file /oracle/app/admin/fmisb/udump/fmisb1_ora_377144.trcThu Dec 1 06:30:05 2011Job queue slave processes stoppedAll dispatchers and shared servers shutdownThu Dec 1 06:30:11 2011ALTER DATABASE CLOSE NORMAL2011-12-8 logProcess q001 died, see its trace fileThu Dec 8 07:58:51 2011ksvcreate: Process(q001) creation failedThu Dec 8 08:21:22 2011Starting ORACLE instance (normal)sskgpgetexecname failed to get nameLICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Interface type 1 en2 168.168.1.0 configured from OCR for use as a cluster interconnectInterface type 1 en0 10.194.0.0 configured from OCR for use as a public interfacePicked latch-free SCN scheme 3Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0SYS auditing is disabledksdpec: called for event 13740 prior to event group initializationStarting up ORACLE RDBMS Version: 10.2.0.4.0.System parameters with non-default values: processes = 1000 sessions = 1105 event = 10298 trace name context forever, level 32 sga_max_size = 16106127360 _shared_pool_size = 10368319488 _large_pool_size = 16777216 _java_pool_size = 16777216 _streams_pool_size = 0 spfile = /dev/rfmis_spfile sga_target = 16106127360 control_files = /dev/rfmis_control1, /dev/rfmis_control2 db_block_size = 8192 _db_cache_size = 5687476224 compatible = 10.2.0.3.0 log_archive_dest_1 = location=/archlog log_archive_dest_2 = log_archive_format = %t_%s_%r.dbf db_file_multiblock_read_count= 16 cluster_database = TRUE cluster_database_instances= 2 thread = 1 instance_number = 1 undo_management = AUTO undo_tablespace = UNDOTBS1 recyclebin = OFF _row_cache_cursors = 40 remote_login_passwordfile= EXCLUSIVE db_domain = dispatchers = (PROTOCOL=TCP) (SERVICE=fmisbXDB) local_listener = LOCAL_FMISB remote_listener = LISTENERS_FMISB job_queue_processes = 4 background_dump_dest = /oracle/app/admin/fmisb/bdump user_dump_dest = /oracle/app/admin/fmisb/udump core_dump_dest = /oracle/app/admin/fmisb/cdump audit_file_dest = /oracle/app/admin/fmisb/adump db_name = fmisb open_cursors = 500 pga_aggregate_target = 1975517184Cluster communication is configured to use the following interface(s) for this instance 168.168.1.71Thu Dec 8 08:21:26 2011cluster interconnect IPC version:Oracle UDP/IP (generic)IPC Vendor 1 proto 2DIAG started with pid=3, OS id=286844PSP0 started with pid=4, OS id=291038PMON started with pid=2, OS id=352548LMON started with pid=5, OS id=307218LMD0 started with pid=6, OS id=311358LMS0 started with pid=7, OS id=241738LMS1 started with pid=8, OS id=278946MMAN started with pid=9, OS id=356456DBW0 started with pid=10, OS id=258364LGWR started with pid=11, OS id=389420CKPT started with pid=12, OS id=282896SMON started with pid=13, OS id=241958RECO started with pid=14, OS id=303600CJQ0 started with pid=15, OS id=303352MMON started with pid=16, OS id=360598Thu Dec 8 08:21:27 2011starting up 1 dispatcher(s) for network address (ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP).MMNL started with pid=17, OS id=274794Thu Dec 8 08:21:28 2011starting up 1 shared server(s) .Thu Dec 8 08:21:31 2011lmon registered with NM - instance id 1 (internal mem no 0)Thu Dec 8 08:21:31 2011Reconfiguration started (old inc 0, new inc 28)List of nodes: 0 1 Global Resource Directory frozen* allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid according to instance 1 * domain 0 valid = 1 according to instance 1 Thu Dec 8 08:21:32 2011 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned outThu Dec 8 08:21:32 2011 LMS 0: 0 GCS shadows cancelled, 0 closedThu Dec 8 08:21:32 2011 LMS 1: 0 GCS shadows cancelled, 0 closed Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues grantedThu Dec 8 08:21:39 2011 LMS 1: 0 GCS shadows traversed, 0 replayedThu Dec 8 08:21:39 2011 LMS 0: 0 GCS shadows traversed, 0 replayedThu Dec 8 08:21:39 2011 Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeLCK0 started with pid=20, OS id=188726Thu Dec 8 08:21:48 2011ALTER DATABASE MOUNTThu Dec 8 08:21:52 2011Setting recovery target incarnation to 2Thu Dec 8 08:21:52 2011Successful mount of redo thread 1, with mount id 4092946819Thu Dec 8 08:21:52 2011Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)Thu Dec 8 08:22:25 2011Completed: ALTER DATABASE MOUNTThu Dec 8 08:22:25 2011ALTER DATABASE OPENPicked broadcast on commit scheme to generate SCNsThu Dec 8 08:22:25 2011LGWR: STARTING ARCH PROCESSESARC0 started with pid=22, OS id=311714Thu Dec 8 08:22:26 2011ARC0: Archival startedARC1 started with pid=23, OS id=299268Thu Dec 8 08:22:26 2011ARC1: Archival startedLGWR: STARTING ARCH PROCESSES COMPLETEThread 1 opened at log sequence 11289 Current log# 4 seq# 11289 mem# 0: /dev/rfmis_redo1_4a Current log# 4 seq# 11289 mem# 1: /dev/rfmis_redo1_4bSuccessful open of redo thread 1Thu Dec 8 08:22:26 2011MTTR advisory is disabled because FAST_START_MTTR_TARGET is not setThu Dec 8 08:22:26 2011ARC1: Becoming the no FAL ARCHARC1: Becoming the no SRL ARCHThu Dec 8 08:22:26 2011ARC0: Becoming the heartbeat ARCHThu Dec 8 08:22:26 2011SMON: enabling cache recoveryThu Dec 8 08:22:30 2011Successfully onlined Undo Tablespace 1.Thu Dec 8 08:22:30 2011SMON: enabling tx recoveryThu Dec 8 08:22:30 2011Database Characterset is ZHS16GBKOpening with internal Resource Manager planwhere NUMA PG = 1, CPUs = 8replication_dependency_tracking turned off (no async multimaster replication found)Starting background process QMNCQMNC started with pid=25, OS id=294994Thu Dec 8 08:22:37 2011Completed: ALTER DATABASE OPENThu Dec 8 08:28:46 2011Reconfiguration started (old inc 28, new inc 30)List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 0 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned outThu Dec 8 08:28:46 2011 LMS 0: 0 GCS shadows cancelled, 0 closedThu Dec 8 08:28:46 2011 LMS 1: 0 GCS shadows cancelled, 0 closed Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IRThu Dec 8 08:28:47 2011Instance recovery: looking for dead threadsInstance recovery: lock domain invalid but no dead threadsThu Dec 8 08:28:47 2011 LMS 1: 138488 GCS shadows traversed, 0 replayedThu Dec 8 08:28:47 2011 LMS 0: 137164 GCS shadows traversed, 0 replayedThu Dec 8 08:28:47 2011 Submitted all GCS remote-cache requests Fix write in gcs resourcesReconfiguration completeThu Dec 8 08:28:51 2011Reconfiguration started (old inc 30, new inc 32)List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned outThu Dec 8 08:28:51 2011 LMS 0: 0 GCS shadows cancelled, 0 closedThu Dec 8 08:28:51 2011 LMS 1: 0 GCS shadows cancelled, 0 closed Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues grantedThu Dec 8 08:28:52 2011 LMS 0: 14094 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14069 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14250 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14234 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 13862 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14064 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14100 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14094 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14264 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14085 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14046 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14251 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14228 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14220 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14137 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14186 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 14088 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 1: 14218 GCS shadows traversed, 4001 replayedThu Dec 8 08:28:52 2011 LMS 0: 10455 GCS shadows traversed, 2955 replayedThu Dec 8 08:28:52 2011 LMS 1: 11426 GCS shadows traversed, 3190 replayedThu Dec 8 08:28:52 2011 Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeThu Dec 8 08:29:47 2011ALTER SYSTEM SET service_names=fmisrac SCOPE=MEMORY SID=fmisb1;Thu Dec 8 08:32:45 2011ALTER SYSTEM SET service_names=fmisrac,fmisb SCOPE=MEMORY SID=fmisb1;Thu Dec 8 08:42:19 2011Memory Notification: Library Cache Object loaded into SGAHeap size 73264K exceeds notification threshold (51200K)二、故障分析 从这两次故障发生现象，客户反馈系统停止响应，动弹不了（用户执行的是“报表模块附加项批量输入”操作）。管理人员去排除故障，发现rac 实例不可用，通过重启实