Loading incremental sources will require full extract from Match360 to consuming systems

For our customer, sources will be loaded into Match 360 gradually. First 5 sources will go thru initial load and then ongoing inbound process will be turned on to apply updates thru ongoing_sync APIs. Match 360 itself is not a point of data consumption, data needs to be extracted and moved into customer consumption environment. We will move data afterh initial load and then ongoing outbound process will deliver updates to the customer consumption environment. When new sources need to be loaded, there are only couple of options: 1) we loaded sources thru ongoing inbound process - that will make sure events are generated on the outbound and customer consumption environment is in sync with Match 360 2) Load new sources thru bulk load (or UI). But that will not generate events on outbound side and in that case the whole population of Match 360 needs to be extracted again and reloaded into customer cunsumption env. Option 1) would be very slow given there are tens of millions records that need to be loaded. 2) Scratching and reloading customer consumption env every time when new source is loaded is not sufficient.

So we need to design a mechanism to extract change events during bulk load of the new source to the existing Match 360 previously loaded sources.

Needed By

Not sure -- Just thought it was cool

Post comment

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Loading incremental sources will require full extract from Match360 to consuming systems