IBM to look for trigger file when placing Cubes in sFTP site to avoid Cube files from being separated during transfer

Originally had this case open: TS001299703

So basically, we are trying to build cubes here in onPrem Transformer, sFTP to IBM so they are available in the CA11 Prod Cloud environment each day. The sFTP steps are proving to be the bottleneck because of the number and size of the cube files.

Currently once the cubes are built, we are to place them in the ‘share' folder of the IBM sFTP site. There is a job in the sFTP site that runs every 10 minutes to pull whatever might be there, into the encrypted disk side of IBM. Once there, the CA11 Prod cloud environment can see them and our data sources, reports, packages can read and use those cubes.

The issue is we must get all the cubes (some are partitioned) before the 10 minute job executed, separating them. If they are separated, we must upload the cube, in its entirety again, otherwise Cognos thinks it's broken. Some of these cubes have 4 files and are 0.5, 0.6 GB in size; they can take a few minutes each to reach the site.

IBM support has suggested we placed them in a directory down from ‘share', in a temp type directory, and once they've all copied, move them up to ‘share' manually. We cannot accept a manual option to perform each day. Especially when we don't really know when those files will be available to copy; it could be 3:30 AM CST or 7:15 am CST. sFTP also doesn't offer a ‘move' command.

We have tried to mitigate this using some scripted methods, but is not full proof; the cubes are subject to being separated.

Thus, the suggestion for IBM to look for a trigger instead of simply executing the job every 10 minutes. We thought, what if, every 10 minutes the job runs and first looks for a specifically named file, CA11trigger.txt. It'll be a blank file always named as such. We'll script that file to be placed only on the condition we have executed our jobs that send our 5 cubes and their files. This way we can be certain the trigger will be placed last and IBM will only execute the intake portion of the job once it spies the CA11trigger.txt file.