Skip to Main Content
IBM Data and AI Ideas Portal for Customers


This portal is to open public enhancement requests against products and services offered by the IBM Data & AI organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:


Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,


Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea


Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.

IBM Employees should enter Ideas at https://ideas.ibm.com


Status Submitted
Created by Guest
Created on Aug 8, 2024

Address Manta log-viewer API timeout performance

Current state:

When a Manta user tries to download a workflow log (to debug or attach to a support ticket), the Manta instance often (in 99% cases) returns with a HTTP 504 Gateway timeout error. This happens because:
1. Manta is connected to an AWS ALB (load balancer) which has a default timeout for 60 seconds.
2. Manta's log viewer API doesnt start responding with any data till it has captured the entire data which could take several minutes.

Increasing the ALB timeout to 5 mins solves the problem for some cases, but most logs still require more than 5 mins to start downloading so this is not a viable solution. Also, we now become dependent on the size of the file - the bigger the file the longer the timeout has to be set

This is a big problem because:

1. The longer the timeout of the ALB, the slower it becomes at detecting problems, and the easier it is to cause a denial of service issue - if a user calls the log-viewer API several times, all the threads of the load balancer and manta itself will become saturated, waiting for a timeout, and other users will get blcoked from doing anything.

2. Billing costs for ALB goes up when threads are busy (doing nothing), which is why the default value of the ALB is set to 60 seconds.

Suggested State:

(1) describe the timeout value that Manta expects from load balancers, so that Manta work.
(2)Reimplement this log viewer API: curl 'https://manta.acme.com/manta-admin-gui/private/log-viewer/v1/workflow-executions/293/export?workflowExecutionId=293&attachTextual=true&attachRealLogs=true' -H 'accept: application/json, text/plain, */*' -H 'accept-language: en-US,en;q=0.9' -H 'authorization: Bearer xyz' -H 'priority: u=1, i' -H 'referer: https://manta.acme.com/manta-admin-gui/app/index.html?' -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' -H 'sec-ch-ua-mobile: ?0' -H 'sec-ch-ua-platform: "Windows"' -H 'sec-fetch-dest: empty' -H 'sec-fetch-mode: cors' -H 'sec-fetch-site: same-origin' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'

Reimplement this to use a STREAMING solution so that the log contents start getting generated while Manta is reading the log file, instead of the current implementation where it waits for the whole file to be read and prepared. You can use the Java streaming APIs to achieve this. See tutorial at https://www.baeldung.com/spring-controller-return-image-file for implementing the stream for spring-web, and https://stackoverflow.com/questions/58309057/efficient-stream-of-zip-files-in-java on how to create a zip file while streaming instead of having to write a zip file onto disk.

Business impact for not fixing this:

1. IBM will be unable to address Manta bugs filed by users, because users will not be able to provide workflow logs.

Needed By Yesterday (Let's go already!)
  • Guest
    Reply
    |
    Aug 12, 2024

    I'm not really familiar with this time efficiency but I'll try to figure it out when I play tiny fishing.