Refresh of SAP SuccessFactors ECP/EC test environment

This month, we’ve been working at PostNL on refreshing their Employee Central Payroll (ECP)/Employee Central (EC) test environment. They have almost 200,000 ‘Central Person’ records, with more than a quarter of a million Employee records linked to them; some of whom are only in ECP – but the majority are also in EC.

The EC Instance Refresh is a service provided by SAP SuccessFactors, which is typically part of your contract, and updates all the data and config from the Production instance in QA or Test. The challenge that goes with that is how to deal with the linked payroll system. A full system copy can be incredibly disruptive in the post-processing required, and means the full disk space of the Production system is needed for the test system. But without this there will be misalignment between the newly refreshed EC instance and the older data in ECP. This is where EPI-USE Labs comes in. There are however choices on how to leverage multiple technologies to solve the problem.

We last did the process about four years ago, and leveraged the Data Sync Manager™ (DSM) Object Sync™ for SuccessFactors Add-on engine, meaning that we simultaneously imported the ECP data and masked the SuccessFactors data at the same time.

This time around we’ve taken a different approach, using the Client Sync™ engine (also part of the DSM Suite) rather than Object Sync, and then the DSM Data Secure™ for SuccessFactors Add-on. Client Sync is typically more of a Basis team capability, which is intended for large- scale changes to systems rather than ‘data on demand’, and is typically used in lieu of a system copy.

There were two reasons for wanting to do it differently:

1. When dealing with high volumes of data, the Client Sync engine is the better solution on the ABAP stack. Object Sync can be used for high numbers, but because it is selecting one record in PA0003, then finding all the related records from all other tables before moving to the next one, it can never be as fast as Client Sync – which processes table by table. You can still be ‘selective’ but only in the sense of not taking the full history of the cluster data (something that was essential on this project given the amount of payroll history).
2. SuccessFactors has been ‘instance refreshed’ so the vast majority of the entities are exactly how we need them. With Object Sync, it was sending the full OData model starting from the User and linking in EmpEmployment, PerPerson, PerPersonal etc. The vast majority of the data was essentially just replacing itself. That of course doesn’t prevent the OData API from running Business Rules and other validations. Last time, we spent a lot of time working around these things to ensure the scrambled data was being successfully upserted. Even if the validation is not a concern, the performance is. 95% of the processing time was the OData API calls so reducing the number would only improve the runtime.

There are still a fair number of preparation steps to consider, as there are just for having the instance refresh, but the aim is that this approach is much more easily repeatable. After an implementation and process definition from our consulting team, it could be something carried out by our clients themselves, or alternatively we can offer a managed service for it.

Technical process: data refresh

We scheduled a Client Sync export to run at exactly the same time SAP SuccessFactors was taking the instance refresh image from the Production SuccessFactors instance. There is a specific ‘HCM only’ profile in the Client Sync product that allows you to update only the HCM data, without affecting data for other areas of SAP. You can optionally include HCM-related customising, but in this use case we do not want to, because the transport management system has been used effectively in ECP.

In this project, it was critical to leverage the ‘time-slice’ capability of Client Sync to only take the last two years of payroll cluster data. Without this, the volume of data would have been too much for the existing QA system.

Once this export was complete, we then began the import on the QA ECP instance. Ideally, you would want the SAP operations team to switch off database redo logs, but if this is not possible, we can just throttle Client Sync by using a lower number of processes. We chanced it with four processes, but there was a point where we'd clearly filled the database logs; but it thankfully recovered unaided. The worst-case scenario would have been a delay waiting for the SAP operations team to clear out some logs, but the performance would be better if they were switched off. This is a long-standing recommendation of ours when it comes to Client Sync, but this is one use case where it can be more flexible, if needed, because the HCM data, preferably filtered on only the last couple of years of transactions, is typically much smaller.

Once the import was complete, we then asked the client to activate and validate replication. At this point, we knew the ‘copy back’ was successful, and now we just needed to move on to scrambling to make sure no sensitive personal data would be available in a testing environment. Did you know that the SAP Data Processing Agreement explicitly states that real data shouldn’t be present in a test system? Paul wrote a blog about this a few years back. There is some masking capability in SuccessFactors, but it doesn’t cover complex data in ECP, like the cluster data, so we needed another part of DSM for this.

Technical process: data scrambling

The DSM Data Secure™ for SuccessFactors Add-on capability came more recently (a few years after Object Sync Success Factors) and extends the ABAP stack Data Secure scrambling engine to be able to read and send back anonymised values to EC via the OData API. So this is the first time we’ve leveraged that for this process. From the refreshed client, we built a work list of Central Person numbers. The masking is driven from this to ensure that if there are multiple employment records (several pernrs for one person) then they will be processed together and get the same values.

We started testing our masking policy with a handful of employees to make sure we were happy with the outcome before we processed too many. The masking simultaneously updates the ECP data, not just Infotypes but also applying the changed names, addresses etc. to the cluster, and now declustered, payroll results. Another advantage of only taking the last two years of payroll history is that there is less data to mask. The work can be split over multiple background processes, and can be resumed if there are HTTPS errors in the OData interface. In this example, bunching the data in 5000 or 10,000 groups seemed to be the optimum, with 4-6 processes for each run.

Invariably, you will hit errors with the OData API and you can end up with data which has been scrambled in ECP but not completely in EC, often including the name. Data Secure uses an ‘if same – keep same’ logic, which means that once something is not aligned, the masking will keep it not aligned.

To counter this, we took an approach which we developed in the previous project of scrambling employees to one fixed value, such as John Doe, in order to align the data – and then reprocessing them with the random name generation option. We also developed a utility that helps to identify all the affected records which will become a standard DSM utility very soon.

The extras

When we last did this, we also built a few utilities to fix up other things that remained after the instance refresh. Links to documents being an important example. The documents themselves are replaced with a dummy document, but the title of the document still persists. Using the EPI-USE Labs library code to build OData requests allowed targeted deletion of the types of attachments that could store sensitive data in the document name, such as ‘Jane Doe’s doctor’s note’, or disciplinary documents.

We also required a photo deletion utility because a handful of employees had gone beyond just a profile picture (which SAP optionally removes as part of the instance refresh) and put their whole background as a personal photo of them. It’s a generational thing…

Then, my favourite extra: before we instigated the first refresh, the organisation maintained a special payroll area which had some data from the original data load which had been adapted and then continued to be maintained so as to provide an easy distinct testing set. Since it was built from partially real keys, it meant the data clashed with real data we were bringing down with Object Sync. Instead of trying to retain the data, we built a custom conversion that can take any employee and move them into this specialist payroll area to form a segregated test set. And of course at any time you can recopy them from Production, with the same masking policy used with Data Secure, to take them out of that payroll area and return them to their real one.

This was our story. How do you do it?

If you have EC and ECP, or even an on-premise payroll linked to EC, how do you deal with updating the ABAP system when you do an instance refresh on EC? I guess for smaller organisations with a few thousand employees, the SAP client copy might work. If your payroll is in a system with full FI/LO data, then you might be falling back on a full system copy.

With DSM, we could avoid a system copy, only bring HCM relevant data and even filter the payroll and time history to just bring the last two years of data. Once you have the data though, what about data privacy? Or do you keep production-like permissions in the test systems at all times?