In my previous post, I have covered how to run ST06, navigate through important ST06 screens and understand those screens. In this post, I would talk about how to use SAP transaction ST06 for performance analysis:
- Use SAP ST06 to do CPU workload analysis
- Use SAP ST06 to do SAP system memory load analysis
- Other analysis
I mainly focus on CPU and Memory analysis which are often showed up as a concern to application/system performance. SAP ST06 itself cannot provide all data needed for CPU load and memory performance analysis. SAP ST06 can be used to find out when and where there is a performance concern related to CPU and memory etc for further analysis. Working in conjunction with other performance tools like SAP workload monitor ST03/ST03N, database load monitor- ST04, SAP transaction statistics tool STAD and SAP snapshots collector -/SDF/MON etc, you can identify SAP work processes , jobs/programs, users who contributed to CPU/Memory performance issue, then solutions/options can be reviewed and worked on subsequently.
1 Use SAP ST06 to do CPU workload analysis
CPU workload analysis is to make sure that CPU workload is not exceeding recommended utilization 80% and maintaining CPU workload balance by analyzing peak load period and load distribution pattern.
1.1 Review CPU load to make sure CPU utilization under the recommended threshold
One of important goals for CPU workload analysis is to make sure that CPU utilization is not over general recommendation – Sum of User and Sys CPU utilization should not over 80% on hourly basis. For application server, system utilization portion of CPU should not be over 10% and for data base server, it should not be over 20%. It is more critical to maintain at least 20% CPU idle rate for database server since database server is a central instance and all SAP transactions/programs need to access database server one way or another.
You can run ST06 and refresh the main screen to see whether CPU is over utilized now. You can use hourly CPU utilization history for the past 24 hours or daily average in the past 30 days provided by SAP ST06 to see whether CPU has been over utilized in the past. If there are situations that those thresholds are exceeded, you need to look into the details on who (jobs, processes or users) are contributing most to the over utilization so you can review the situation and identify the solution.
Then you might ask how to identify top jobs, processes or users who is consuming a lot of CPU power. Unfortunately, SAP ST06 itself cannot provide SAP jobs or processes names. You need to use different SAP or non SAP tools to complete the performance analysis dependant on whether you are reviewing a current CPU load concern or past CPU load concern. Since we are dealing with SAP system performance and the CPU overutilization is mostly related to processes from SAP in most cases, this post would cover details how to identify top SAP jobs, processes or users. There are cases when non SAP processes are making significant contribution to CPU overutilization, I would briefly mention this.
1.1.1 How to identify TOP CPU SAP user, processes, programs/jobs related to current CPU load issue
First, you need to refresh ST06 “Top CPU” process screens to know which ST06 process is a top CPU process. Following is a part of ST06 “Top CPU processes” screen:
Figure 1 ST06 TOP CPU processes
If a process is continuously showed at top of above screen during the observation period, then it is an expensive process. If it is related SAP work process or a SAP database session/process. You can find corresponding sap user, sap work process or database session, SAP job/program name via following method.
For a process in ST06 Top CPU processes screen which is related to a SAP work process, you can use following way to identify related SAP user, program or jobs-
- Get the “Process ID” from ST06 Top CPU processes screen.
- Run SAP work process monitor SM50 and locate SAP work process based on “Process id”. From SAP SM50 screen, you can know the SAP process type, program name and SAP userID.
- For SAP background process, you can find corresponding SM37 job name: Run SE16 against SAP table TBTCO, put “process-id” into field TBTCO-WPPROCID, execute it, you would get SM37 job name. There are other ways to get background job name based on “process ID”.
For a process in ST06 Top CPU processes screen which is related to a SAP database process, you can get the corresponding database process and further the SAP work process if applied:
- Get the “Process ID” from ST06 Top CPU processes screen.
- Run SAP ST04 database process/session monitor. “Process ID” in step 1 is actually “Operating System PID” in ST04 session monitor. So you can identify the corresponding database session to know the SQL statement which is causing the load on the database server from there, you can review the SQL statement to identify any possible improvement opportunity. Please refer to my post on how to tune SQL.
- Most of database processes are result of running SAP application jobs or programs. You can identify the corresponding job or programs via client ID in ST04 session monitor screen. Client ID is actually mapped to SAP50/SM66 work process ID.
- You can identify corresponding SAP job if it is a background process (see above).
If a process in ST06 Top CPU processes screen is a non-SAP process and you do not have access to operating system shell, you need to talk with your system-level administrator to find out operating system level program/script/job name.
ST06 process name like DW.* is related to SAP work process in SM50. ST06 process name like “ORACLE*” is related to a database session/process in ST04. This has been covered in my previous post how to run ST06, navigate through important ST06 screens and understand those screens.
1.1.1 How to identify TOP CPU SAP user, processes, program/jobs related to past CPU load
For past CPU overload, you need to use SAP transaction ST03/ST03N SAP system workload monitor to analyze the workload for the specific period to identify top offenders. SAP transaction ST03/ST03N is an important performance tool. You can use ST06 data as an input for ST03/ST03N analysis. If /SDF/MON is scheduled in the system, you can use /SDF/MON system performance snapshots to do further analysis.
1.2 Review CPU load to achieve better load balance
Another goal of CPU workload analysis is to achieve CPU load balance including vertical and horizontal balance. Vertical balance is to make sure that the same SAP server has proper load balance over time period such as a day or a week. The objective of vertical CPU load review is to spread CPU load out over time so a server CPU power can be used equally in different time period as much as possible. The objective of horizontal CPU load review is to spread CPU load among all servers/instances of a SAP system so each server of a SAP system has equal utilization at any specific moment whenever possible. In reality, it is impossible to achieve absolute vertical or horizontal CPU utilization balance due to various reasons business office hours and non business office hours, business pattern(seasonal ), different level of resource consumption by different programs/jobs and different level of resource consumption by the same program due to volume difference etc.
SAP ST06 can display hourly CPU utilization for past 24 hours. It can display daily average CPU utilization for a single server and across systems for the past 30 days. This information is an input for load balance analysis. SAP ST06 cannot tell you which users and processes are creating those loads. You need to work with SAP workload monitor ST03/ST03N to find further details.
Load balance issue is normally caused by many concurrent running SAP work process as a result of parallel solution, immediate processed EDI/ALE interface in my experience and incoming RFC calls etc. SAP CPU workload balance analysis should look into those areas first. It is really rare that culprit is singe thread of SAP work process.
1.3 Understanding CPU load and application/system performance
CPU load is an important factor for application/system performance. But there is no simple equation to say: high CPU load = bad performance, low CPU load = good performance; High CPU load = bad user experience, Low CPU load = good user experience. You can refer to my previous post to understand other factors which might influence a program performance. However I would assume here that CPU is only factor which influence performance to make discussion simple.
Even hourly average CPU utilization is under 80%, this does not mean all business transactions executed in that hour would have normal performance. Whether there is a performance impact, this really depends on CPU load distribution in that hour and how sensitive a business transaction is to runtime. For example, if CPU utilization for 1st half hour is 90% and 2nd half hour utilization is 10%, then average CPU utilization is 50% for that hour well below the recommended utilization, however performance of a business transaction which is running in the 1st half hour would be very likely impacted. This would also depends on normal utilization If normal utilization is 50%, then CPU utilization jumps to 80%, an application or transaction performance might be impacted. If a job/program normally runs for several days, then single hour CPU over utilization on runtime of such job or program can be ignored. In another way, when we evaluate CPU impact on a long running transaction/job runtime, we need to check CPU load history instead of current CPU load.
Also if a SAP system has separate database server and application server, one server has overloaded but another server is ok.. the performance impact on a transaction might depends on where the transaction spends most time. If 99% of time is spent on database server and database server is not over loaded , CPU contention on an application server is very likely have no impact on the performance of transaction where it is executed on application server. So on.
Even performance of all business transactions in the SAP system are impacted from technical analysis point view. This might be different from what business users feel. Since each business transaction and process has an acceptable runtime range that is SAP users’ expectation as well. Business users would only feel the performance impact if business transaction or performance has a notable deviation from its’ normal runtime range. For example, a background job has a run time range of 1 2 hours based on volume, business might not feel difference if what should be finished in 1 hour normally for small volume takes 1 and half hour due to CPU overutilization. However, business would feel the performance issue if big volume happens to kick in while there is a CPU overutilization in this case.
Technically, CPU load would impact a program/job performance if a business job needs CPU but have to be put to wait due to CPU contention. When a job/program spends a lot of time waiting for CPU, its’ performance is impacted. SAP transaction STAD would show statistical data of job/program runtime which break job runtime down to further components. Analysis of those components can help you to understand the impact of CPU contention. For example, in server with 4 processors running only 4 jobs and each of 4 jobs is purely doing calculation w/o disc for hours, You should see CPU 100% utilization but each of 4 jobs spends no time in waiting for CPU.
Under current SAP technology, we normally do not need to worry about individual CPU/processor utilization showed in Snapshot CPU screen (Not TOP CPU screen which shows a list of work processes). However, this could be a concern if one or more of server CPUs are reserved for specific purpose/application.
1.4 Other CPU workload analysis tools
There are other tools for CPU load analysis. I have mentioned SAP workload monitor SAP transaction ST03/ST03N in previous section. It can be used to do workload analysis as well. You can use ST06 to identify peak load period and use SAP workload monitor ST03/ST03N for further analysis. SAP database load monitor ST04 can be used to monitor SQL operation load on database server. I would cover SAP workload monitor ST03/ST03N and database load monitor ST04 in the future.
My favorite tool for load monitor analysis is SAP performance snapshot program /SDF/MON. /SDF/MON is not good for current CPU contention but it can be used to collect CPU load snap-shots for historical utilization analysis. ST ST06 is good for current CPU load analysis but depends on other tools to find process, sap job, user and program information. So you can ST06 to know when there is a CPU contention, then use SAP /SDF/MON transaction to collect snapshot for that particular period for details analysis. You can refer to my post SAP /SDF/MON workload analysis for details on how to use SAP transaction /SDF/MON in load analysis.
1.5 Process relation among SAP ST06, SM50/SM66 and ST04
This section would tell you the linkage among ST06 TOP CPU process, SM50 SAP work processes and ST04 database process/sessions via examples. Exception where a process exists in one tool but not in another tool due to SAP job cancellation etc is not discussed here.
1.5.1 How is a ST06 database related process linked to a SAP SM50 work process
Figure 2 ST06 DB related process -> SM50 process
1.5.1 How is a ST06 SAP-related process linked to a SM50 SAP work process
Figure 3 ST06 OS process -> SM50 process
1.6 Possible solution for mitigating CPU work load and achieving better load balance
Through above analysis, you identify the top peak period and top business transactions/jobs, you can consider following solutions to reduce CPU load and achieve better load balance:
- Schedule run t job in different time when system is less busy or change job frequency
- Tune job/program logic and database operation So the job/program is more efficient and use less CPU. This can include job variant change.
- Workload balance/redistribution run the job/program in another server/instance of the SAP system.
- Upgrading hardware more powerful server or more CPUs/Processors.
Upgrading hardware should be the last resort after all other options have been tried. For tuning ABAP program performance, you can refer to my post how to do performance trace using SAP ST12 and how to tuning ABAP program performance if you like.
If external process is using a lot of CPU power, then you can use similar solutions as what we have for SAP related processes. Or you can simply terminate it and stop it depending on your scenario.
2 Use SAP ST06 to do Memory load analysis
Memory workload analysis is to make sure that there is enough free memory and free swap space (Virtual memory) available in the system by analyzing peak memory and memory usage pattern.
SAP ST06 can show you current server memory utilization and hourly average memory utilization for past 24 hours. Please refer to my post how to run ST06 on ST06 screen navigation and understanding. Following is a part of past 24 hours memory history screen:
Figure 4 ST06 – past 24 hr memory history
To know current SAP server memory utilization status, you can just run ST06 transaction and refreshing the main screen every 10 seconds. If you would like to know which SAP process/user is consuming the server memory, you need to access mode list screen of SAP transaction ST02 SAP buffer/memory monitor. ST06 provides top CPU processes but unfortunately it cannot provide Top memory processes.
Figure 5 ST02 mode-list
To know history memory usage, you can run SAP ST06 screen and navigate to past 24 hours memory screen. However, if you observed a particular period when the memory usage is a concernthat is all you can get from SAP ST06. You can use this information to schedule SAP transaction /SDF/MON to get memory snapshots for the period concerned, Then based on /SDF/MON memory snapshots, you can figure who and which processes are top memory consumption processes/users for the period in questions. That is why SAP /SDF/MON stands out as one of my favorite performance tool both for SAP production support and SAP development. Please refer to my post on how to run /SDF/MON for memory analysis if you would like to know more on it.
Figure 6 /SDF/MON snapshots
Memory load analysis also pays attention to swap space. I normally just check whether there are enough free swap space and hourly paging out rate. Hourly paging out rate for UX and Linux OS should not be over 20% of physical memory in general.
Here I assume that all memory consumption in a server are from SAP users or sap processes. SAP ST02 and SAP /SDF/MON is for analyzing memory consumption from SAP users in the instance/system where it is started. If there are other instances on one server or there are operating system level processes which consume a lot of memory, they would not be captured by SAP ST02 or SAP /SDF/MON. UX level or other sap instances need to be checked at the same time.
Possible solutions to mitigate performance concerns due to memory overload and achieve better SAP memory load balance are:
- Reduce volume you might need to educate end business users or change job variant.
- Schedule Run job/transaction in different time when there is less memory load.
- Redeployment run the job in different instance or server.
- Tune the SAP job/program to reduce memory usage.
- More capacity add in additional physical memory or swap space.
As for CPU load analysis, adding more capacity should the last resort when other options have been reviewed. If external process/task is causing issue, you need to work with owner and look into solution depends on actual scenario.
4 Other analysis Via SAP ST06
Based on ST06, we can do IO analysis based on and file space analysis. Objective of this analysis is to make sure that there is no hot spot or IO bottleneck by proper IO configuration such as spreading data out among system disc properly. You can refresh ST06 Main screen and pay attention to DISC with highest response time to observe whether a particular disc has high utilization for a significant duration and has a lot of process in waiting (queue length). You also can check past 24 hours disc usage history to find out whether a particular disc has an extremely high utilization and queue length. I have not run into such situation in my SAP performance work so far. No further insight can be shared. SAP system IO configuration is not my area as well.
Figure 7 -ST06 Disc with Highest response time
File space analysis is to check that we have enough free space at OS directory level. This might be critical for successful running of jobs/programs which need to output file to operating system level.
4. Further clarification
Up to now, I have covered on how to use SAP operating system monitor ST06 to do performance analysis: Identify existing/potential problems using SAP operating system monitor ST06 and continue ST06 analysis with other SAP performance tools. From ST06, you can access other SAP tools such as network traffic check which can be handy for troubleshooting network related performance issue.