Spark job to be able to access, read and write on external database without the credentials being exposed anywhere (instance, logs, job metadata, etc…)

We want to create a job on a Spark instance of the Analytics Engine powered by Apache Spark that runs a Spark application reading from and writing to an Exasol database. To enable database access, we need to pass our user credentials into the job.

Our challenge is that so far, we haven’t found any way to provide these credentials without exposing them somewhere — whether in the instance itself, in the logs, on the history server, or within the job metadata.

The KMS documentation mainly covers encryption of Parquet files and key lifecycle management, which doesn’t address how we need to handle database credentials in Analytics Engine jobs. The section about using secrets from vaults in connections works well for CP4D connections in projects and notebooks, but in our case, Spark jobs don’t have access to connection assets in Projects.

The general vaults overview talks about integration with external vaults, but we don’t have access to external vaults. It also doesn’t give us a concrete way to safely inject secrets into AE jobs without exposing them in logs or metadata. Additionally, the CP4D vault itself isn’t directly accessible from inside a Spark job, because that would require passing a token into the job — which again exposes sensitive information in the job metadata.

What we really need is clear guidance on how to supply credentials (or tokens) to AE jobs in a way that they are not stored in the job metadata. Passing them via arguments or environment variables still exposes the credentials in job metadata, so that’s not a viable solution for us.

With the current system behaviour, our data analysts are not allowed to use a Spark instance for the analysis of data in data bases. That is a significant limitation for us.