Data Manager enhancement: Mark job as DONE after file reaches final destination

We want the main job to be marked as DONE only after the data file has been successfully transferred to the user-specified destination. Currently, with the integration of LSF and Data Manager, the main job is marked as DONE as soon as the 'bstage out' process completes. However, this only indicates that the output file has been copied to the staging area cache — not to the final destination specified by the user.

Needed By

Month

Post comment

Admin

Bill McMillan

Jul 14, 2025

There are two basic methods of moving data in LSF.
1) bsub -f "a>b" -f "c<d" a.out : this moves the data in-band as part of the job.
2) bsub -data "dataspec" a.out : this moves the data out-of-band before and after the job runs using data manager.

The request is essentially for data manager to perform the stage out of data in an "in-band" rather than "out-of-band" method.
Changing the core logic for DataManager to be in-band would remove the key benefit of data manager - not tying up idle compute resources while waiting for data movement.
As such, we don't think implementing would have general product value.

There are a couple of alternative approaches:
a) the transferid's can be obtained from the job, and the external script can bwait() on the transferid's

b) you could combine both transfer methods:
$bsub -data inputdataspec -f “outputfile <” a.out
where the input data would be staged by data manager in an out of band manner, and the results returned in-band.

Reply
Hide replies

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Data Manager enhancement: Mark job as DONE after file reaches final destination