Trouble-shooting SAP job program.docx

Trouble-shooting SAP job/program termination on memory errorPosted on 2013年08月04日 星期日 by eric It is not unusual that a SAP job/program can be terminated on memory error. In my experience, I have seen SAP job/program terminations on following memory errors: TSV_TNEW_PAGE_ALLOC_FAILED, TSV_TNEW_BLOCKS_NO_ROLL_MEMORY, TSV_TNEW_OCCURS_NO_ROLL_MEMORY and SYSTEM_NO_ROLL. This post would share my experience on how to do trouble-shooting on single sap Job/program termination due to memory errors. It would cover: 1. How to trouble-shoot job/program termination on memory error, 2. Case 1 TSV_TNEW_PAGE_ALLOC_FAILED due to server/instance memory contention, 3. Case 2 TSV_TNEW_PAGE_ALLOC_FAILED due to exceeding SAP memory quota for individual work process, 4. Cases TSV_TNEW_BLOCKS_NO_ROLL_MEMORY,TSV_TNEW_OCCURS_NO_ROLL_MEMORY and SYSTEM_NO_ROLL 5. General recommendation on how to fix SAP job/program termination on memory error, 6. Further clarification 1 How to trouble-shoot job/program termination on memory error1.1 understanding SAP memory area and allocation sequence for individual SAP process/programsA running SAP process needs memory to store user context like program internal table etc. There are several SAP memory areas from which the memory requirements of user programs are satisfied. For technical reasons, approximately the first 250 KB of a user context is always stored in the roll area, additional data of the program is stored according to following sequence based on SAP design for dialog work process. 1. to the limit ztta/roll_first in the roll area, then 2. In extended memory, to the limit ztta/roll_extension or, if extended memory is exhausted, then 3. In the roll area again, until the roll area is full, then 4. In the heap memory (PRIV mode), to the limit abap/heap_area_dia or abap/heap_area_total or until the address space or the swap space is exhausted. In the case of non-dialog SAP work processes, the sequence 1- 3 - 4 - 2 applies. After this, process is terminated with error if the process needs more memory. SAP users online activities like creating customer order using va01 transaction are typical tasks which would engage SAP dialog work process. Remote Function Call or ALE task use SAP dialog work process as well. A SAP SM37 Job would engage SAP background work processes. Depends on design and business function, additional SAP work process type like SAP updating work process can be involved or triggered as a result of user online activities. The same is true to a SM37 Job. 1.2 Trouble shooting process for termination on SAP memory errorWhen a SAP program is aborted on memory issue, this could be due to: 1. Exceeding SAP memory quotas for individual work process and 2. SAP memory contention on SAP server (one time or regular), 1. SAP memory configuration (individual quota, SAP memory areas) and 2. Sever memory bottleneck (physical memory + swap place). To know which reason is applicable to a particular job termination, following are general steps 1. Find out How many memory is used by the job/program at time of termination via ST22 or /SDF/MON at time of termination, 2. Review SAP memory quota for individual SAP work process and SAP memory area configuration via ST02, 3. Comparing memory usage by the job/program against SAP quota and SAP memory usage 1. If memory usage at time of termination is less than memory quota, then termination is mostly due to SAP memory contention and 2. If memory usage at time of termination is close to memory quota (both extended and heap memory) at time of termination, then termination is mostly due to hit memory ceiling specified by SAP setting. 2 Case 1 TSV_TNEW_PAGE_ALLOC_FAILED error due to SAP memory contention2.1 Job aborted due to memory error TSV_TNEW_PAGE_ALLOC_FAILEDSAP job log via SAP SM37 shows that job was cancelled on memory error TSV_TNEW_PAGE_ALLOC_FAILED (this is a BW system). Figure 1 SM37 runtime memory error TSV_TNEW_PAGE_ALLOC_FAILED2.2 SAP memory usage by the job SAP ST22Run ST22 to find specific entry related to your job cancellation. Following is related to the above job. Figure 2 -ST22 Memory usage at time of job termination2.3 SAP system/Instance memory configuration SAP ST02You can use SAP ST02 transaction to find SAP system/instance memory quota for single process. Please make sure that you look into the memory quota for the instance where job termination happened. Following is SAP memory quota on the instance where above job was executed. Figure 3 SAP ST02 memory quota for single process2.4 Comparing memory usage by the job against quota Following table is showing the comparison between what is used by the job and what is allowed by SAP memory configuration. Table 1 SAP job memory against quota System memory contentionMemory areaMemory consumed by the job(bytes)SAP memory quota(KB)CommentRoll11,682,88011,719Heap121,160,16010,00,00Heap memory contention Extended2,002,728,2241,953,125Based on above table, 121MB heap memory at time of abortion was used by the job while system quota for single job is 10G. This normally means that available SAP memory was occupied by other processes/jobs in the server/instance. In another word, memory contention at that time was a direct cause for the termination. If SAP /SDF/MON is scheduled to collect SAP memory snapshots, you can use that to verify SAP extended memory, SAP heap memory, SAP roll memory usage and SAP Page memory at system/instance level close to the moment of SAP job/program termination. ST22 would not tell SAP memory usage situation at instance/system level. SAP ST02 shows memory usage high-water mark since start of instance which might be helpful in some situation. 3 Case 2 TSV_TNEW_PAGE_ALLOC_FAILED due to exceeding SAP memory quotas3.1 Job aborted due to memory TSV_TNEW_PAGE_ALLOC_FAILEDFollowing screen shows a SAP job was terminated at error TSV_TNEW_PAGE_ALLOC_FAILED. Figure 4 SAP Job termination on memory error3.2 Memory usage by the job SAP ST22Run ST22 to find specific entry related to your job cancellation. Following is related to the above job. Figure 5 ST22 memory usage at time of termination3.3 SAP memory quota for jobPlease refer to above figure 3. 3.4 Comparing memory usage by the job against quotaFollowing table is showing the comparison between what is used by the job and what is allowed by SAP memory configuration. Table 2 SAP Job termination on memory error memory usage against quotaMemory areaMemory consumed by the job(bytes)SAP memory quota(KB)CommentRoll687904011,719Heap10,235,210,54410,00,00Up to quotaExtended2,002,728,2241,953,125Up to quotaFrom above table 2, you can know that SAP job should be asking more memory than what is allowed for a single SAP work process. So the SAP job was terminated. 4 SAP program termination on TSV_TNEW_BLOCKS_NO_ROLL_MEMORY, TSV_TNEW_OCCURS_NO_ROLL_MEMORY or SYSTEM_NO_ROLL4.1 SAP process termination on TSV_TNEW_BLOCKS_NO_ROLL_MEMORYFollowing case is likely due to SAP memory contention Figure 6 Termination on error TSV_TNEW_BLOCKS_NO_ROLL_MEMORYFigure 7 ST22 core-dump on TSV_TNEW_BLOCKS_NO_ROLL_EMMORY under quotas4.2 SAP process termination on TSV_TNEW_OCCURS_NO_ROLL_MEMORYFigure 8 shows memory consumpt Figure 8 core-dump on TSV_TNEW_OCCURS_NO_ROLL_MEMORY over quotasFigure 9 core-dump on TSV_TNEW_OCCURS_NO_ROLL_MEMORY under quotas4.3 SAP process termination on SYSTEM_NO_ROLLFigure 10 and 11 are two core-dumps on SAP memory error SYSTEM_NO_ROLL from the same server. Core dump showed in Figure 10 was due to process using memory over quota and Figure 11 was related to server level memory contention. Figure 10 core-dump SYSTEM_NO_ROLL over quotasFigure 11 core-dump SYSTEM_NO_ROLL under quotasI would like to point this out again that SAP transaction /SDF/MON can be used to analyze Memory contention at instance level When memory from SAP heap memory area is used up, it is more likely to see process termination. Please refer to Figure 12, At 12:45:00, Heap memory usage at server level is over 27G while total heap memory is 28G so you can see several core dumps at 12:46:01 and 12:46:02. Figure 12 /sdf/mon and ST22 cross-check5 General recommendation to fix job termination on SAP memory errorIf system/instance memory contention is a cause for the job/program cancellation, you can avoid the error via one or several of following solution depending on your specific case: 1. Tune SAP instance memory parameters to give more memory to one or more SAP areas. Normally we need to increase SAP heap memory and extended memory. But it could be page memory, SAP Global extended memory etc. If there is no more free “physical” memory at the sap server, then you need to bring more memory. 2. Rerun or reschedule run it when system memory usage is low or in different time window or you can reschedule top memory offender job to different time window. Top memory user is not necessary the same job which has been aborted. 3. Reduce volume of a SAP job should process via program variant etc. This could be other jobs other than aborted job since other job could use a lot of memory causing other job failure. If exceeding memory quota is a cause for the SAP job/program cancellation, you can consider one or more of following options to avoid error: 1. Reduce volume of a job should process, 2. Increase SAP memory quotas for a single job or adjust corresponding SAP memory parameter and 3. Tune program to reduce memory usage. Depending on your specific situation, a SAP job might still use too much memory even when memory usage of a job is not hitting the server/instance memory limit. Before you can increase memory allocation to SAP memory area, you need to make sure that there is free memory available in the system you can monitor system memory via SAP transaction ST06. 5 Further clarificationSAP memory allocation/parameter changes via SAP RZ11 transaction come into effect after system restart. However SAP does provide a program RSMEMORY to allow you to change individual SAP memory quotas etc. with immediate effect without restart of your SAP system. I would cover this in my future post. In this post, I am assuming that information report