Enhancement Request: Provide End-to-End Audit Traceability for Spark Application Stop Events in watsonx.data

We would like to request an enhancement to improve auditability and traceability for Spark application stop events in watsonx.data.

Currently, when a Spark application transitions from RUNNING to STOPPED, the available logs do not provide sufficient information to determine the exact initiator of the stop action. This makes it difficult to identify whether an application was stopped by a user, service account, automation process, API call, platform policy, or an internal platform operation.

Requested Enhancement

Provide detailed audit information for Spark application stop events, including:

• Initiating user ID or username
• Service account identity (if applicable)
• Calling application or automation process
• API endpoint associated with the stop request
• Request ID / Correlation ID
• Source IP address or originating service
• Timestamp of the stop action
• Spark application ID and job ID
• Reason for the stop action

Desired Event Attribution

For every Spark application stop event, provide clear attribution indicating whether the stop was initiated by:

• Interactive user action
• Service account action
• API invocation
• Scheduled automation
• Platform policy or governance action
• Internal watsonx.data platform operation
• Administrative action

Use Case

While investigating an unexpected Spark application termination, we observed the following:

• Application state changed from RUNNING to STOPPED
• Platform logs showed TERM and kill signal processing

However, we were unable to determine:

• Who or what initiated the stop action
• Whether a stop API was invoked
• Which identity performed the action
• Whether the stop originated from a user, service account, automation process, or platform operation

Business Value

Providing stop-event attribution would:

• Reduce troubleshooting and root cause analysis time
• Improve operational visibility
• Improve audit and compliance capabilities
• Reduce dependency on IBM Support for stop-event investigations
• Enable teams to independently identify who stopped an application and why

Expected Outcome

Customers should be able to determine, through audit logs or monitoring interfaces:

Who stopped a Spark application, when it was stopped, how it was stopped, and the originating user, service, API, or platform action responsible for the stop event.

This enhancement would significantly improve supportability and operational transparency for Spark workloads running in watsonx.data.