Runtime Logging Export for DataStage

Administrators, operations teams, and DataStage users need the ability to export DataStage runtime logs to an external storage service, such as ADLS Gen2, while a DataStage flow or pipeline is actively running.

Currently, there is a way to export runtime logs to an external service after the DataStage flow or pipeline run has completed. However, users need near real-time or in-flight log export during execution, rather than only post-run export. This capability is important for operational monitoring, troubleshooting, governance, and broader platform adoption.

The requested enhancement is to provide a supported method to stream or continuously export DataStage runtime logs during the execution of a DataStage flow or pipeline to an external destination such as ADLS Gen2 or other supported external log storage services.

This feature should support the ability to:

• Export runtime logs while the DataStage job, flow, or pipeline is still running.

• Stream or periodically write log updates to an external storage location.

• Support ADLS Gen2 as a target destination.

• Preserve job run identifiers, timestamps, project names, flow names, pipeline names, and execution status metadata with the exported logs.

• Allow operations and governance teams to analyze log history outside of the Cloud Pak for Data platform.

• Provide configuration options at the project, job, flow, pipeline, or platform level.

• Support secure authentication and authorization to the external storage destination.

• Ensure exported logs are complete, ordered, and traceable to the corresponding DataStage run.

• Support long-running jobs where waiting until completion to access externalized logs is not practical.

This enhancement would help teams perform operational analysis, monitor in-flight execution behavior, troubleshoot issues earlier, and maintain centralized log history outside of the CPD platform. Having runtime logs available in external containers during execution would also support governance use cases, historical analysis, auditability, and integration with downstream monitoring or reporting tools.

Providing real-time or near real-time DataStage runtime log export would improve user adoption of the platform by making it easier for target users to manage, monitor, and govern DataStage workloads using their existing external log analysis processes.