WAS宕机常见问题及参考解决方案.doc

WAS宕机问题总结起来有以下几类：一、线程挂起导致线程池满（Thread Hang）:Hangs refer to the JVM locking up or refusing to respond.A hang can occur when:1）Your application entered an wait leak2）Excessive Synchronization cause performance problems3）A deadlock has occurred收集日志：生成JAVA CORE分析工具：IBM Thread and Monitor Dump Analyzer1、线程等待泄漏(Wait Leaks)常见的情况就是，有很多线程使用wait()方法，等待被唤醒notify（）。但是，存在某一个线程获取到锁并且进行业务处理完成后，忘记去唤醒等待执行的进程。这样导致的等待泄漏，从而线程挂起。When you use the wait/notify mechanism, you typically have one or more threads blocked in the wait() call, waiting to be notified. The notifying thread is supposed to call notify() or notifyAll() to signal that waiting threads can wake up and carry on processing.A problmatic situation could occur that the notify() call is invoked before the threads go into the wait() method. In this case, the waiting threads would not be notified anymore and become stuck. So, do not forget to notify the waiting theads in you application and maks sure that the Notify action is performed after all the waiting threads are started. 应对策略：notify必须发生在所有wait之前。2、同步过度（Excessive Synchronization）LIS系统FORMULA ONE报表打印就是采用了SYNCHRONIZE思想，如果有某一个大的报表长时间打不出来的话，会导致其它所有的报表打印线程全部挂起。Synchronization is required for protecting some critical resources, but if large sections of code are synchronized, an application effectively will become single threaded, and the throughput will decrease dramatically. To improve the performance, you should think about:1) Use other efficient algorithm to replace the monitor/lock?2) Try to keep the monitor lock for a short time?3) Reduce the number of threads using this monitor/lock?应对策略：改变程序实现算法，避免占用锁时间过长。比如设置多个锁或者提高同步代码块的执行效率。3、资源抢占死锁（Dead Lock）线程运行需要获取到两个资源，当线程1获取到资源A，线程2获取到资源B，此时，线程1需要再获取资源B，线程2需要再获取资源A。这就形成了资源循环申请的一个死锁。其它想获取到资源A或者资源B的线程必将都会挂起。In this case, a thread acquires the lock on lock1, At the same time, another thread acquires the lock on lock2. so the threads are deadlocked: neither thread will give up its lock until it acquires the other lock, but neither will be able to acquire the other lock until the other thread gives it up. One of the best ways to prevent the potential for deadlock is to avoid acquiring more than one lock at a time, which is often practical. However, if that is not possible, you need a strategy that ensures you acquire multiple locks in a consistent, defined order. 应对策略：申请资源锁，按顺序申请，或者一次性全部申请。二、CPU使用率高（High CPU Usage）:High CPU usage problem refer to that a system is performing poorly because of lack of CPU resources.A high CPU usage problem can occur when:1）Some threads consuming a lot of CUP resources2）A frequent GC probelm occurs1、程序自身耗用高CPU（比如：Infinite Loop）收集日志：生成JAVA CORE分析工具：操作系统监控工具（比如TOPAS PS之类的） IBM Thread and Monitor Dump Analyzer应对策略：调整程序，改变算法，使得最终耗用较少CPU资源。2、频繁垃圾回收（Frequent GC）多次循环，去申请很多的小对象。从而需要垃圾回收器去频繁释放很小的内存空间。It is recommended that users enable the VerobseGC first and the Pmat or GCVM will be helpful to find out the frequent GC.Heap dump tells you the objects info of java heap when it is triggered. You can use Heap dumps to debug the java heap issues. You can press the Tool - Generate Dump menu to triger a heap dump which will show objects information. 收集日志：打开详细垃圾回收，生成HEAP DUMP分析工具：(IBM Support Assistant)Memory Dump Diagnostic for Java (MDD4J)应对策略：调整垃圾回收策略，或者改变算法，调整程序，把很多局部变量（小对象）全局化。三、内存溢出（Out Of Memory）:The JVM maintains two memory areas, the Java heap, and the native (or system) heap.These two heaps have different purposes, are maintained by different mechanisms, and are largely independent of each other. OutOfMemory condition that occurs could be due to either running out of Java heap or Native heap. 1、本地内存溢出（Native OOM）收集日志：生成系统宕机日志（system dump） OS monitor tools, such as Svmon, top,vmstat and so on分析工具：（IBM Support Assistant）Dump Analyzer 应对策略：暂时无。2、finalize执行时间过长（Blocked in Finalize）程序员自己写finalize的处理代码，导致执行时间过长，从而影响JVM的高效垃圾回收处理策略。最终内存被占满。The Java service team recommends that applications avoid the use of finalizers if possible. The JVM specification states that finalizers are for emergency clear-up of, for example, hardware resources. The service team recommends that you use finalizers for this purpose only. Do not use them to clean up Java software resources or for closedown processing of transactions. Because it will affect the garbage collection, and may cause a memory problem.Finalizers are an expensive use of computer resources and they are not dependable.The Java service team does not recommend that you use finalizers for process control or for tidying Java resources. In fact, use finalizers as little as possible.For tidying Java resources, consider the use of a cleanup routine. When you have finished with an object, call the routine to null out all references, deregister listeners, clear out hash tables, and other cleanup operation. Such a routine is far more efficient than using a finalizer and has the useful side-benefit of speeding up garbage collection. The Garbage Collector does not have so many object references to chase in the next garbage collection cycle.收集日志：生成HEAP DUMP 分析工具：(IBM Support Assistant)Memory Dump Diagnostic for Java (MDD4J)应对策略：使用设置不再使用的HASHMAP之类的变量为NULL，以便垃圾回收器更快回收此类对象。3、内存泄漏（Java Memory Leak）当你要用到公共的HASHMAP变量，不断地往里ADD对象，但没有remove操作。这个时候就发生有内存泄漏，时间长了，内存就被耗光。The Java heap becomes exhausted when garbage collection cannot free enough objects to make a new object allocation. Garbage collection can free only objects that are no longer referenced by other objects, or referenced from the thread stacks.Java heap exhaustion can be identified from the -Xverbosegc output where the garbage collections are occurring more and more frequently, with less memory being freed.收集日志：生成HEAP DUMP 分析工具：(IBM Support Assistant)Memory Dump Diagnostic for Java (MDD4J)应对策略：标记那些不再使用的对象为NULL，从根本上去杜绝或避免内存泄漏的发生。4、大对象内存溢出（Large Object OOM）In this case, there is no memory leak, but some large objects are allocated and it will increase the memory usage significantly and may cause OOM errors. So, do not forget to release the objects which will never be used in your application, especially for some large objects, it will affect the performance significantly.收集日志：生成HEAP DUMP 分析工具：(IBM Support Assistant)Memory Dump Diagnostic for Java (MDD4J)应对策略：牺牲IO，拆分大对象为N个小对象，以减少对内存的使用。或者调整参数，设置大对象区域。四、连接池满（J2CError）:When using a j2c, it is important to follow specific programming practices to prevent problems. Care should be taken to make sure resources are released even in exception condition. Resource exhaust may cause performance impact or even deadlock.1、连接泄漏（Connection Leak）中科软LIS系统的经验告诉我们：（A、)连接泄漏可能存在以下几种情况：（1）schema的问题（2）PubSubmit自身连接泄漏，在某些异常处理的时候没有释放连接。（3）自己申请数据库连接，自己处理事务，自己释放连接。（B、）解决办法：（1）使用新的MAKE工具，保证Schema数据库底层操作不会连接泄漏。（2）严格避免自己去申请数据库连接，自己处理数据库SQL语句。（3）调整PUBSUBMIT，充分避免在提交数据库操作异常的时候，有连接不释放的情况发生。Applications can suffer from performance problems and even appear to “hang” if they do not close their connections properly. This is most often caused by developers not properly using the connection.close() method. To ensure that connections will be closed properly, they should be closed in a “finally” block. WebSphere Application Server is instrumented to eventually time-out orphaned connections and return them to the pool, but for an application that makes frequent use of database connections, this might not be enough. New connections can get queued up waiting for the database while old connections are waiting to be timed out. This can bring the application grinding to a halt, and you can see connectionWaitTimeoutExceptions. When using a connection pool, it is important to follow specific programming practices to make connection pooling work most efficiently and to prevent problems. One of the most important practices is to always call close() on a connection object as soon as the application finishes using it. This is necessary so that the connection can return to the free connection pool. Care should be taken to make sure connections are closed even in exception condition. Failing to close connections means that connections are leaked, reducing the number of connections available in the connection pool, and leading to ConnectionWaitTimeoutExceptions. 收集日志：To enable connection leak diagnostics, set the Log Detail Level to “ConnLeakLogic=finest” and enable diagnostic tracing. 分析工具：Tivoli Performance Viewer应对策略：找到连接没有关闭的地方，将其关闭。2、连接长时间被占用（LTC Connection Occupation）典型表现：尽管数据库SQL语句执行很快，但 SERVLET/JSP 本身仍未完成，则这个数据库连接依然对其它线程失效，标记为不可使用。所以说，如果有很多长时间的SERVLET/JSP，应该排队挨个处理。When a servlet/jsp is invoked, the Web container in WebSphere Application Server automatically