Enable DPH to Register, Search, Share, and Subscribe to Unstructured Data and Its Vectorized Representation

Provide a capability in DPH to register both:

• Original unstructured data stored in an S3-compatible Object Storage

• Processed/vectorized unstructured data stored in OpenSearch after parsing, chunking, and embedding

Both types of unstructured data should be available as Data Products in DPH, allowing users to:

• Search and discover unstructured data through semantic search.

• Share unstructured Data Products with other users.

• Subscribe to unstructured Data Products.

• Apply the same enhancement approach to future DPH capabilities for unstructured Data Products (Example: displaying custom properties for Columns in the table, supporting actual data samples, etc.)

Currently, users cannot directly search and discover the unstructured data through the Object Storage. In addition, the OpenSearch team confirmed that achieving this type of user experience would require a separate interface to be built on top of OpenSearch.

Therefore, the requested capability is to provide this functionality directly through DPH, enabling users to search, share, and subscribe to unstructured data in the same manner as structured Data Products.

For reference, the customer is currently building an on-premises data platform based on CPD 5.4. Plus, their goal until early next year is to build a DPH-based portal where all users can semantically search, share, and subscribe to both structured and unstructured data.
Furthermore, the ultimate goal is building a knowledge graph across structured and unstructured data.