IBM Data Platform Ideas Portal for Customers


This portal is to open public enhancement requests against products and services offered by the IBM Data Platform organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:


Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,


Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea


Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.

IBM Employees should enter Ideas at https://ideas.ibm.com



Add a new idea Subscribe
Status Needs more information
Workspace Data Lineage by Manta
Created by Guest
Created on Nov 14, 2024

RELATED IDEAS

Some Manta workflow finish status "FINISHED SUCCESSFULLY" in Admin UI and REST API, yet the logs indicate errors. The status should reflect data quality issue

Why is it useful?

Current Manta 42.4.4+ APIs Does not meet API standards and conventions. If this is fixed, Manta API consumers would benefit who are primarly responsible for automating data lineage use cases. In the process, it would improve code quality of Manta product and ease the work who are leveraging the manta platform for their usecases.

Currently, Manta does not have capability to schedule jobs. The workaround provided is to use custom script that leverages Manta orchestration APIs and use some existing platforms to schedule it like CRON or some third party tools.

Who would benefit from it?

Consumers of the Manta orchestration APIs who works on automating the data lineage use cases for any organization.In the process, it would improve code quality of Manta product and ease the work who are leveraging the manta platform for their usecases.

How should it work?

We tried to leverage the MANTA orchestration API in our custom code. However, the response of the API doesn't reflect the actual status of the job. We have found that for SAS and IFPC scanners although the job gets complete with "FINISHED SUCCESSFULLY" when we see the logs. We find ERRORs like "INPUT_STRUCTURE_ERRORS" when parsing the IFPC workflows or "PROCESSING_INPUT_FAILED" , "INCLUDE_MACRO_PROCESSING_FAILURE", "MACRO_VARIABLE_VALUE_NOT_FOUND" etc in SAS workflows.

We expect the API response should have some indication that it has data quality issue.
Example: API response could return a boolean status on dataQualityStatus: true/false based on the data parsing errors found during execution of SAS and IFPC scanners.

Reference

https://www.ibm.com/docs/en/manta-data-lineage?topic=ui-manta-orchestration-api

SAS-workflow_wi...
SAS-workflow-st...
SAS-workflow_with_errors.png
Open full size
SAS-workflow-status-FINISHED-SUCCESSFULLY.png
Open full size
SAS-workflow_with_errors.png SAS-workflow-status-FINISHED-SUCCESSFULLY.png
Needed By Not sure -- Just thought it was cool
  • Guest
    Sep 3, 2025


    I have a few questions

    1. how to handle pagination in this API.

    manta-admin-gui/private/log-viewer/v1/workflow-executions/{workflow execution ID}/metadata

    (There could be cases where SAS/COBOL scan can have errors more than page size).

    1. how I can get unique set of error category and issue types for scanners in Manta. Is this scanner specific?


    Reply
  • Guest
    Jun 6, 2025

    There is a good set of API listed as "private" APIs on one of the AdminUI swagger pages.

    The url is here:
    /manta-admin-gui/swagger-ui/index.html?urls.primaryName=%20%20manta-private-log-viewer-api#/

    The private logviewer APIs all begin with
    manta-admin-gui/private/log-viewer/v1/
    all but one of them are GET functions, and they query the log db the same way the log viewer does.

    The most relevant would likely be:
    manta-admin-gui/private/log-viewer/v1/workflow-executions/{workflowID}/metadata
    which returns a json payload of the workflow results including error types, errors, and information.

    Reply
  • Guest
    Mar 10, 2025

    As discussed in a call with Caroline F (3/6), one potential solution that would be to hint to users which "successful" workflows need further triage. If the status table had a column which listed out the count of warning and error log messages, that would hint to the users to drill further into those workflows, at least to review the logs. If this requires changes to the connectors themselves, then we request that the following copnnectors be prioritized first: IFPC, Oracle, SAS, DB2, SQL Server, Netezza, SAP HANA. 

    Reply
  • Guest
    Jan 30, 2025

    Hi Jakub,

    The ask was to get workflow status along with data quality status in a single REST API endpoint. And we are using this endpoint : /public/process-manager/v1/executions/{executionId}/status

    The output for one of the SAS workflow from obove endpoint is


    ... {

    "scenarioName": "sasDataflowScenario",

    "connectionName": "TPAD-Wholesale-21-file",

    "executionStatus": "SUCCEEDED",

    "executionStart": "2025-01-30T17:00:37.229Z",

    "executionEnd": "2025-01-30T17:02:47.780Z",

    "executionProgress": 100

    },

    ... trimmed for clarity...


    From Manta Admin UI,

    It displays data quality issue. For "SAS Dataflow Scenario" it shows 68.5% ; on mouse-hover it shows below stats

    Script processing success rate : 51.7%

    Statement processing sucess rate: 68.5%

    Total script processed: 267

    Total statements processed: 7968



    Open questions:

    1. Why UI vs endpoint have different results for same workflow execution ID.

    2. How the automation will detect if there were any data quality issue using above endpoint?

    Thanks,

    Amar Gurung




    Reply
  • Admin
    Jakub Moravec
    Jan 30, 2025

    Hi team,

    Thank you for your idea! The job scheduling capabilities are available on IBM Manta Data Lineage (Manta running natively on Cloud Pak for Data). There are currently no plans on introducing the capability to the current Manta product.

    With regards to semantics of the workflow status - the FAILED status is reserved for errors that will make the whole workflow fail. The errors shown on the screenshot are not of that character, they typically cause a single input (e.g. a single SAS script, or even a single statement within that script) not to be process, but lineage still can a typically is created for the other inputs that are processed. That being said, there are couple of options how to get more insight into lower-severity erros

    /public/process-manager/v1/executions/{executionId}/status endpoint provides not only status of the workflow, but also status of each step.

    All the details can be obtained by exporting logs for the specific workflow exection using /public/log-viewer/v1/export endpoint. The output is not a best fit for automation purposes hower.

    Please let me know if any of these approaches will help you and if you have further questions.

    Kind regards,
    Jakub Moravec
    Manta Data Lineage Team

    Reply

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.