IBM Data and AI Ideas Portal for Customers


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea

Help IBM prioritize your ideas and requests

The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The product management team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.

Receive notification on the decision

Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.

Additional Information

To view our roadmaps: http://ibm.biz/Data-and-AI-Roadmaps

Reminder: This is not the place to submit defects or support needs, please use normal support channel for these cases

IBM Employees:

The correct URL for entering your ideas is: https://hybridcloudunit-internal.ideas.aha.io


Status Under review
Created by Guest
Created on Jun 18, 2021

WML multiprocessing support

Multiprocessing using Pythons multiprocesing library works with Notebooks in Watson Studio, but not for deployed WML models.

Multiprocessing for DOCPLEX MILP models can significantly reduce time and resources used for solving a set of several models simultaneously. This could e.g. be a set of sensitivity analysis on a model, testing the influence of changes in input parameters.

Brief problem description

  • When deploying a model using the multiprocessing.Pool(f) class, subsequent deployment jobs fail with the error message "PicklingError : Can't pickle <function f at 0x7f0043023200>: attribute lookup f on __main__ failed"

  • I have been in contact with a "IBM Data and AI Senior Customer Advocate, Decision Optimization", who confirms that the pickling error is a limitation in the current WML environment.

4.5x speed increase: an example from our own model

We can solve one DOCPLEX model using a 16 vCPU and 64 GB system in a Watson Studio Notebook in 40 sek. Solving the same model on the same system 8 times using multiprocessing allocating 2 cores pr. solve only increases the solve time to 69 sek. It should be no surprise, solving the same model 8 times sequentially takes around 320 sek.

Hence multiprocessing reduces the time it takes us to explore 8 sensitivities by a factor of around 4.5 in our test case.

Deeper rationale for multiprocessing instead of sequential solve

Most DOCPLEX MILP models doesn't see a linear decrease in solve time, when they are assigned more resources. With internal testing of our own DOCPLEX models, we reduce solve time to about half going from 1 to 4 CPU cores, while increasing to 16 cores only gives a 10 pct. further decrease in solve time. Similar results are found broadly in the MIPLIB2010 benchmarking test-set as seen in http://plato.asu.edu/talks/informs2018.pdf . This limited benefit of many cores is not unique to CPLEX, and also seen in competing solvers.

Therefore, if one has a set of MILP models to solve on e.g. a 16 core system, there is a great benefit of doing this in parallel with multiprocessing, allocating only a few cores pr solve instance, instead of using all cores for each model solve, and solving sequential.

In addition, overhead from manipulating data in Python, and building the DOCPLEX model prior to solving is primarily a single threaded task, which can see a large speedup from multiprocessing as well.

Needed by Date Sep 30, 2021
  • Guest
    Jun 18, 2021

    Thanks for the suggestion. While we are evaluating it, let me offer some alternatives...

    An alternative to multiprocessing within a single WML job is be to create separate WML jobs for each of the optimization models that you would like to run. You could use the WML Python client at https://ibm-wml-api-pyclient.mybluemix.net.

    Or you could use a DO experiment and create multiple scenarios using the Decision Optimization Python client as described in https://medium.com/@AlainChabrier/decision-optimization-python-client-3f6e6c662f6b. This gives you the benefits of input/output handling and visualisation offered by DO for each run, instead of only for the main job.