We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:
Post your ideas
Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,
Post an idea
Upvote ideas that matter most to you
Get feedback from the IBM team to refine your idea
Help IBM prioritize your ideas and requests
The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The product management team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.
Receive notification on the decision
Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.
Multiprocessing using Pythons multiprocesing library works with Notebooks in Watson Studio, but not for deployed WML models.
Multiprocessing for DOCPLEX MILP models can significantly reduce time and resources used for solving a set of several models simultaneously. This could e.g. be a set of sensitivity analysis on a model, testing the influence of changes in input parameters.
Brief problem description
When deploying a model using the multiprocessing.Pool(f) class, subsequent deployment jobs fail with the error message "PicklingError : Can't pickle <function f at 0x7f0043023200>: attribute lookup f on __main__ failed"
I have been in contact with a "IBM Data and AI Senior Customer Advocate, Decision Optimization", who confirms that the pickling error is a limitation in the current WML environment.
4.5x speed increase: an example from our own model
We can solve one DOCPLEX model using a 16 vCPU and 64 GB system in a Watson Studio Notebook in 40 sek. Solving the same model on the same system 8 times using multiprocessing allocating 2 cores pr. solve only increases the solve time to 69 sek. It should be no surprise, solving the same model 8 times sequentially takes around 320 sek.
Hence multiprocessing reduces the time it takes us to explore 8 sensitivities by a factor of around 4.5 in our test case.
Deeper rationale for multiprocessing instead of sequential solve
Most DOCPLEX MILP models doesn't see a linear decrease in solve time, when they are assigned more resources. With internal testing of our own DOCPLEX models, we reduce solve time to about half going from 1 to 4 CPU cores, while increasing to 16 cores only gives a 10 pct. further decrease in solve time. Similar results are found broadly in the MIPLIB2010 benchmarking test-set as seen in http://plato.asu.edu/talks/informs2018.pdf . This limited benefit of many cores is not unique to CPLEX, and also seen in competing solvers.
Therefore, if one has a set of MILP models to solve on e.g. a 16 core system, there is a great benefit of doing this in parallel with multiprocessing, allocating only a few cores pr solve instance, instead of using all cores for each model solve, and solving sequential.
In addition, overhead from manipulating data in Python, and building the DOCPLEX model prior to solving is primarily a single threaded task, which can see a large speedup from multiprocessing as well.
Do not place IBM confidential, company confidential, or personal information into any field.