My customer needs to migrate their regional ECC systems from Unix-based systems to Linux-based system. Some of regional systems are very big and their database sizes are over 10 Terabytes. To reduce system down time and mitigate risks of system migration cutover, the customer decided to use SAP’s Minimized Downtime Service (MDS) to migrate those ECC systems. I am the single point of performance contact for the migration project. If you are going to use SAP MDS to migrate your SAP system and wonder
- Whether SAP MDS would impact SAP performance and how big is the impact?
- What can do to mitigate negative performance impact of SAP MDS.
You might be interested in this post.
1 SAP MDS Brief
When a SAP system is down, it would not be able to process business transactions. So it is important for business operation to reduce migration downtime. SAP MDS is used to meet such customer need by migrating majority of data if not all from source system to targeted system while the source system is under normal business operation. In a high-level, SAP MDS would take a clean database copy while SAP system is up, It would take some time for SAP migration tool to migrate the database copy to targeted system, meanwhile, “new” changes (new records or changes on existing record) from ongoing business operation are generating, new changes are captured via database triggers, MDS would transfer those changes to the targeted system after completing migration of initial copy(clone) – this is called “synchronization”. In this way, when we need to switch from old system(source system) to new system(target system), number of database records needed to migrate to target system is very minimal if not all records are transferred, targeted synchronization level is 99% before downtime, so downtime needed to migrate a sap system from one environment to another environment is greatly reduced as well as associated risk. SAP MDS is an incremental migration solution.
MDS is using database trigger and log tables to capture incremental changes. MDS solution would create a log table and create a table trigger for each database table by default. Changes(Delta) on a table are captured via database trigger and recorded in its’ respective log table for later synchronization.
2 Would SAP MDS impact sap system/application performance?
It is stated that MDS would impact the Oracle overall performance in a minimal way. But in my experience, we do see negative performance impact both technically and end-business impact after SAP MDS is enabled in our system.
2.1 SAP MDS – negative impact on system performance
Based on database performance data technical analysis, MDS solution does has impact on overall system performance. Database triggers were enabled in our production system on Sep 6 as part of MDS solution. Following chart is showing weekly average response time for each sequential read and change(micro seconds/operation). It clearly shows that there is a database performance deterioration after MDS is active.
How would this impact business transaction run time?
2.2 SAP MDS – Negative impact on application performance
Following chart is screen copies of transaction profile from two weeks – Week of Aug 11 – 17(left) when SAP MDS was not enabled and week of Sep 8 – 14(right) which MDS was active.
From above chart, you can clearly see that database time per transaction step in MDS week is noticeably higher for top 12 reports/transactions – this contributed to longer response time per transaction step for all top 12 reports/transactions except for transaction VA01 and ZVOMH. For VA01, average database time is higher in MDS week but offset by much lower CPU time which leads to better response time in MDS week. Better performance for ZVOMH in MDS week might be due to much lower business volume – CPU time per transaction step in MDS weeks is 404 ms/step which is much smaller than 878 ms/step pre-MDS week.
In average, response time per transaction step is 10-50% higher in MDS week. Impact on individual jobs/program would vary since each program has different function, design and code.
You can use SAP transaction ST03N to do application performance comparison analysis such prior upgrading and after upgrading etc. You can use SAP ST03N transaction to get the transaction profile showed as above.
2.3 SAP MDS – Negative impact on individual background jobs
If a program/transaction runs 10-50% longer, would this impact business operation? This depends on gap between business performance requirement and program/job performance position. For example, if requirement is that a job has to finish in 1 hour and the job max duration is 10 minutes prior to enable MDS… Then even the job takes up to 59 minutes in MDS, it is just a technical impact on job runtime which has no business impact. Business might not even notice this runtime increasing.
Following table shows average run time and maximum run time for a list of daily jobs (they are executed once daily) in two periods of two weeks from our production environment..
Job Average Run-time (sec)
|Business background Job||Pre-MDS (2 weeks)||Post-MDS(2 weeks)||Pre-MDS (2 weeks)||Post-MDS(2 weeks)|
Above table shows that SAP MDS would make a job average run time 13% – 150% longer and make maximum job run time about 6 times longer! Jobs in the above table run in sequence – they are critical path of a job chain. Business has a cut-off deadline for the last job at the last row of above table. Since each preceding job was running longer after SAP MDS was active, accumulated effect on the last job is big – this resulted in a deadly miss to the cut-off time.
So now, you have seen data that SAP MDS can impact performance, how can we avoid this? What can we do to mitigate the negative performance impact from SAP’s MDS?
3 How can avoid SAP MDS’s negative impact on SAP performance?
It is stated that SAP MDS has to create database trigger for catching the changes and log tables for storing the changes. Each change would be stored in log table in addition to normal transaction table. And MDS would need to read the log table and transfer it to the target server (synchronization) via parallel processes.
MDS activities would use some amount of system resource. Make sure that your system has enough system resource (CPU, Memory, Disc space, network bandwidth) and sufficient SAP work processes to take care of MDS activities.
Prior to MDS, we should tune system setting to cate for additional objects and load from MDS:
- SAP memory tuning such as SAP Nametab buffer etc. catering for additional ABAP table object,
- Database buffer and redo/log space for additional table object and additional reading/updating on log tables,
- Identify top changes tables and their jobs,
- Remove the top changed tables from MDS solution – especially number of changes is high and table size is small. You do not want to remove a huge table from MDS solution – it might need more downtime to copy it over – against objective of SAP MDS,
- Design parallel process schedule used in SAP MDS data synchronization schedule according to system load. Changes are transferred to targeted system by using parallel processes. Number of parallel processes used in data synchronization can impact application performance. It is better to use less number of parallel processes in performance critical period while more parallel processes can be used in data transfer in non-critical period. For example, SAP MDS is allowed to use up to 10 parallel processes for data synchronization in critical period. 35 parallel processes in business hours. In other period, we allow MDS to use 45 parallel processes and
- Hourly monitoring was also built to monitor system based on pre-established performance KPI like log file synchronization etc.. So SAP MDS activities can be adjusted timely to avoid negative impact on performance.
From application point view:
- Review critical jobs and their performance position – identify potential victim of MDS activities,
Take action to address potential performance concern on potential victim before SAP MDS is enabled in your system:
- Schedule change – start it earlier,
- Remove un-necessary dependency,
- Speed the process by breaking down the original volume and engaging parallel processing such as multiple jobs,
- Suspend non-critical jobs which are doing massive database changes and
- Monitor critical application performance closely after SAP MDS is active so potential performance issue can be detected earlier for action.
In the above case, the performance issue was reported in mock migration. We addressed the issue by removing one step out of chain since analysis indicates there is no absolute business need to build that job into the job chain. We also split the last step job into 3 jobs. Disable table trigger on the most-changed table by the program, redesign plan of parallel processes needed in SAP MDS data synchronization stage. …With those changes, the performance issue was addressed and the job was able to finish prior to the cut-off time during the 2nd migration mock as well as migration phase.