This portal is to open public enhancement requests against products and services offered by the IBM Data & AI organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).
Shape the future of IBM!
We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:
Search existing ideas
Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updateson them if they matter to you. If you can't find what you are looking for,
Post your ideas
Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,
Post an idea
Upvote ideas that matter most to you
Get feedback from the IBM team to refine your idea
Specific links you will want to bookmark for future use
Multiprocessing using Pythons multiprocesing library works with Notebooks in Watson Studio, but not for deployed WML models.
Multiprocessing for DOCPLEX MILP models can significantly reduce time and resources used for solving a set of several models simultaneously. This could e.g. be a set of sensitivity analysis on a model, testing the influence of changes in input parameters.
Brief problem description
When deploying a model using the multiprocessing.Pool(f) class, subsequent deployment jobs fail with the error message "PicklingError : Can't pickle <function f at 0x7f0043023200>: attribute lookup f on __main__ failed"
I have been in contact with a "IBM Data and AI Senior Customer Advocate, Decision Optimization", who confirms that the pickling error is a limitation in the current WML environment.
4.5x speed increase: an example from our own model
We can solve one DOCPLEX model using a 16 vCPU and 64 GB system in a Watson Studio Notebook in 40 sek. Solving the same model on the same system 8 times using multiprocessing allocating 2 cores pr. solve only increases the solve time to 69 sek. It should be no surprise, solving the same model 8 times sequentially takes around 320 sek.
Hence multiprocessing reduces the time it takes us to explore 8 sensitivities by a factor of around 4.5 in our test case.
Deeper rationale for multiprocessing instead of sequential solve
Most DOCPLEX MILP models doesn't see a linear decrease in solve time, when they are assigned more resources. With internal testing of our own DOCPLEX models, we reduce solve time to about half going from 1 to 4 CPU cores, while increasing to 16 cores only gives a 10 pct. further decrease in solve time. Similar results are found broadly in the MIPLIB2010 benchmarking test-set as seen in http://plato.asu.edu/talks/informs2018.pdf . This limited benefit of many cores is not unique to CPLEX, and also seen in competing solvers.
Therefore, if one has a set of MILP models to solve on e.g. a 16 core system, there is a great benefit of doing this in parallel with multiprocessing, allocating only a few cores pr solve instance, instead of using all cores for each model solve, and solving sequential.
In addition, overhead from manipulating data in Python, and building the DOCPLEX model prior to solving is primarily a single threaded task, which can see a large speedup from multiprocessing as well.
Do not place IBM confidential, company confidential, or personal information into any field.