IBM Data and AI Ideas Portal for Customers


This portal is to open public enhancement requests against products and services offered by the IBM Data & AI organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:


Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,


Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea


Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.

IBM Employees should enter Ideas at https://ideas.ibm.com


Add a new idea Subscribe
Status Planned for future release
Workspace Data Lineage by Manta
Created by Guest
Created on Feb 9, 2023

RELATED IDEAS

Kafka and S3 interfaces

Lack of support for Lineage Metadata Import within CP4D for S3 (or other object storage services)

Lack of support for Lineage Metadata Import within CP4D for Kafka (furthermore, I don't even think Kafka is a supported connection type at all in CP4D)

Lack of support for "Confluent Cloud Schema Registry and other Kafka distributions apart from Confluent on-premises"

And then some additional questions are:
 

Can Manta currently support lineage scanning for objects in S3 (particularly table objects), even if it all has to be done outside of CP4D UI?

Can Manta currently support lineage scanning for Kafka even if it all has to be done outside of CP4D UI (I believe the answer is yes on this)?

If Manta can support lineage scanning for Kafka, is HTTP basic authentication the only supported authentication method that manta can use to access schema registry?

To what extent can lineage in Manta be manually updated? And can manual lineage entries be added even for sources that manta doesn't have scanners for?

Would still like some information and/or enablement on how the scanners are actually used for each of the main technologies that AAP has mentioned and what information is extracted by the scanner for each technology.

Needed By Quarter
  • Admin
    Jacob Stellon
    Reply
    |     Apr 3, 2023

    MANTA recently released a scanner for S3. The S3 scanner connects to each configured Amazon S3 instance, scans all the buckets, filters them, and then analyzes the contents of all filtered buckets. Customers can use the S3 scanner today but it has not yet been integrated into WKC.

    MANTA also has a Kafka scanner today that is planed to be integrated into WKC in CPD 4.7. MANTA can either connect to Confluent Platform’s Schema Registry and extract the schemas contained in Kafka topics in an automated way or allow the user to describe the Kafka environment on their own to visualize it and benefit from integrations with other scanners. The Kafka visualization includes objects such as a cluster, topics, schemas, and columns. MANTA today does not support cloud based confluent.

    I've attached the Kafka scanner documentation here.


    Manual lineage is supported in WKC via the creation of custom assets and the use of our business lineage APIs to create lineage relationships. A similar feature is available directly in MANTA to create lineage for sources that are not supported


    I recommend watching these enablement sessions for more information: https://ibm.box.com/s/kxx0xfb6jbn2b3sxo67447z4r2so62ws


    image.png
    image.png
    image.png
    Open full size
    image.png
    Open full size
    image.png image.png

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.