IBM Data Platform Ideas Portal for Customers
This portal is to open public enhancement requests against products and services offered by the IBM Data Platform organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).
Shape the future of IBM!
We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:
Search existing ideas
Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,
Post your ideas
Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,
Post an idea
Upvote ideas that matter most to you
Get feedback from the IBM team to refine your idea
Specific links you will want to bookmark for future use
Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.
IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.
ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.
IBM Employees should enter Ideas at https://ideas.ibm.com
This is helpful, I will talk more with Carmen Ruppach (OM handling data connection/ingestion, among other areas) and create an issue. We have coming enhancements to parquet, but this issue is distinct.
Right- Let me explain:
We must be able to recognize multiple files in a folder structure or in a bucket as if they were a *single* data asset, rather than (or in addition to the possibility of) having each file be a separate data asset.
We need this because we want to be able to enter metadata only and access policies once for all files that comprise a data source.
Specific examples:
a) Currently, the ECI (contracts) project needs to be able to store all contracts for a particular geo in a folder. And that geo's contracts can be seen as a data asset. Frankly, the set of geo folders should also be seen as a data asset of all contracts.
b) We want to be able to ingest and store Webhose news items in folders by date. The whole Webhose news is a data source, comprised of folders that each contain multiple files.
We really need to have a data asset that is a set of folders where each folder contains multiple parquet files. This is because a typical logical data asset is comprised of multiple tables.
In terms of use case, one might have a structured data asset that provides N tables. One might put each in a separate bucket to at least have table level access control. Or one might place a smaller collection of K highly related tables into a bucket, in cases where it would never make sense to access one without the others. Within that bucket, one might place multiple parquet files into each folder representing each of the K tables. Each parquet file might represent the records appended during a certain period of time, such as a daily ingestion from a source external to the data lake.
Note that the parquet files may actually span more than a single bucket, which is partially where the requirement stems from. Does this help?
Raj, this seems to be more of a complex issue. We have Folders coming as a Data Asset type to Catalog and Projects, but this does not satisfy your request. Can you explain a bit more how this would help you be more productive and the priority in your opinion?