PySpark Kernel unable to read/extract when connected to external database using Kerberos Keytab

IBM JRE supplied as part of Sym 7.3.1 + Conductor 2.5 is incompatible with current use-case

Use-case: We have a client that uses Spark/Notebooks (PySpark kernel) to connect an external database via ODBC using kerberos. Whenever they try to connect to external DB using IBM supplied versions of JRE (see below) - they are encountering

"unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule"

It is supposed to look for com.ibm.security.auth.module.Krb5LoginModule instead but that isn't the case.


[egoadmin@cbbenp13-cc017 ~]$ cd /opt/middleware/symphony/jre/
[egoadmin@cbbenp13-cc017 jre]$ ls -ltr
total 20
drwxr-xr-x. 3 egoadmin cbb 4096 May 11 12:20 8.0.6.0
drwxr-xr-x. 3 egoadmin cbb 4096 May 11 12:22 8.0.6.20


Both these versions (8.0.6.0 , 8.0.6.20) do not work for current clients use-case when we point the JAVA_HOME to the associated jre


Upon on checking with Support team on this issue, we were provided with OpenJDK (see below) - now, they are able to read data properly however encounter errors when they try to see the content. We need to understand what really happens behind show() that involves Kerberos

[egoadmin@cbbenp13-cc017 ~]$ cd /opt/middleware/symphony/jre/
[egoadmin@cbbenp13-cc017 jre]$ ls -ltr
total 20
drwxr-xr-x. 8 egoadmin cbb 4096 Jun 18 2020 jdk1.8.0_261 --- Provided by Ding from support team

df_table.show() throws an error

" Error: The service in process is not supported. Unable to find valid kerberos
principal for authentication"


This is odd to me, as the Spark read works , but when we view the dataframe it doesnt work - See below

(Below snippet works fine)

jdbcurl = "jdbc:oracle:thin:@//iaasn00095176.svr.us.jpmchase.net:6136/CPA_DEV_DP109649_180221220944_J_READWRITE"


df_table = spark.read.format("jdbc").option("url",jdbcurl) \
.option("dbtable", "(SELECT * FROM sys.dual)") \
.option("oracle.jdbc.allowedLogonVersion","8") \
.option("oracle.net.kerberos5_mutual_authentication","true") \
.option("oracle.net.authentication_services","( KERBEROS5)") \
.option("Driver","oracle.jdbc.OracleDriver") \
.option("oracle.net.kerberos5_cc_name","/cpa/fs1/deploy/keys_ea/krb5cc_309936") \
.load()

We also need your help to understand what happens behind .show()