Feature Request for IBM Knowledge Catalog: Support for Metadata Import from AWS S3 with Deeply Nested Parquet Partition Structures

We have a data source stored on Amazon S3, and we have successfully established a connection to it within the IBM Knowledge Catalog (IKC) environment. The data is stored in Parquet format, and the cloud team has ensured that all required permissions are in place, including ListBucket, GetObject, and KMS (if decryption is required).

Issue Encountered

Assume the following:

· Connection path = a

· S3 bucket = b

· Logical table to import metadata for = C

During metadata import, when selecting the scope, IKC displays the table path as:

/b/C

However, the internal structure of table C inside S3 is significantly more complex. It follows a hierarchical, multi-level partition design, for example:

C/

└── year=2024/

└── month=10/

└── day=01/

├── part-00001.parquet

├── part-00002.parquet

└── ...

└── year=2023/

└── month=01/

└── day=01/

├── part-00001.parquet

├── part-00002.parquet

└── ...

└── year=...

...

In this structure:

· A single logical table corresponds to a root folder (C).

· Under that root folder, there are multiple nested folder levels (e.g., year, month, day).

· Inside each partition folder, there are many Parquet files, and the number of files is not fixed.

Given this reality, the current IKC metadata import mechanism—which requires pointing directly to /b/C—is not able to properly interpret or ingest metadata from such deeply nested table structures.

Request

We kindly request that the product and engineering teams design and implement enhanced functionality that allows IBM Knowledge Catalog to:

· Import metadata directly from AWS S3 even when the data source consists of:

o Simple standalone Parquet files or

o Complex, multi-level nested partition structures

· Recognize a logical table using only the top-level directory path (e.g., /b/C), without requiring manual navigation of nested partitions

· Automatically interpret and parse partitioned folder structures

· Successfully infer schema and metadata regardless of:

o The depth of nested folders

o The number of Parquet files (which may be dynamic and large)