LSF should store accounting data in database

The lsb.acct log file format is anachronistic and hard for customers to deal with. To get the LSF accounting data into a relational database customers have to write their own code to parse lsb.acct files (ether via custom code such as perl/python, or tedious use of C API lsb_geteventrec() calls) and push the positional fields into corresponding database fields. SlurmDBD functionality makes slurm look pretty appealing as a replacement for LSF, just for this reason alone! https://slurm.schedmd.com/accounting.html
With slurm you can simply tell it to log all accounting data to a MySQL or MariaDB database and not have to write/maintain unsupported, custom code and nightly jobs to parse and [Deleted] in lsb.acct data.
Will IBM update LSF for the 21st century in this regard, or will customers like us have to look at other options to replace LSF such as slurm, since our HPC job accounting data is so important to us?

Post comment

Guest

Reply
| Aug 5, 2018

From our own testing with competitive products, the performance of the scheduler is ultimately limited by the speed of writing to the database. Similarly, we have many clients executing millions of jobs per day, and writing millions of rows every day rapidly results in a very large and slow database.

Our design decision was to load the accounting data into elastic search which is much better suited to handling this volume and type of data.

0 reply Hide replies

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

LSF should store accounting data in database