Provide built-in automated lifecycle management and cleanup for Spark logs, temporary files, and residual runtime artifacts generated by IBM Knowledge Catalog workloads.

IBM Knowledge Catalog (IKC) relies heavily on Spark-based processing for Metadata Enrichment and other data discovery/governance workloads. In large enterprise environments with frequent and concurrent enrichment executions, Spark can generate a significant volume of logs, temporary files, runtime artifacts, and residual data.

At Bradesco, one of IBM’s major financial services customers in Brazil, the IKC environment has experienced operational incidents involving Spark-related storage consumption and residual processes/artifacts. During troubleshooting, orphan Spark processes had to be manually cleaned up to restore infrastructure stability. After cleanup, IBM Knowledge Catalog, Analytics Engine powered by Apache Spark, WKC Search, and the OpenShift environment returned to a healthy state, and Metadata Enrichment jobs were initially able to execute successfully again.

However, under increased Metadata Enrichment workload, storage pressure reappeared. A PVC used by the environment reached 100% utilization (100 GiB used out of 100 GiB) even after having previously been increased from 50 GiB to 100 GiB. New enrichment jobs then remained submitted/running without making progress.

This creates an operational dependency on manual troubleshooting and cleanup activities that should ideally be handled automatically by the product.

The proposed enhancement is to introduce a product-managed lifecycle and cleanup mechanism for Spark-generated operational data, including where applicable:

• Automatic cleanup of Spark logs after job completion based on configurable retention policies.
• Automatic removal of temporary files and runtime artifacts that are no longer associated with active workloads.
• Detection and cleanup of artifacts left by failed, interrupted, or orphaned Spark jobs.
• Configurable retention by age and/or storage utilization threshold.
• Protection of artifacts belonging to active jobs to prevent unsafe cleanup.
• Storage utilization monitoring with warnings before critical thresholds are reached.
• Administrative visibility showing what is consuming Spark-related storage.
• Supported/manual “cleanup now” capability through the product UI or supported administrative interface.
• Auditability of automated cleanup operations.
• Recommended/default lifecycle policies appropriate for Metadata Enrichment workloads.

The objective is not simply to provide a cleanup script or workaround, but to make Spark storage lifecycle management a native and supported IKC operational capability, reducing the need for administrators or IBM Support to intervene directly at the infrastructure layer.