Update (or better yet remove) the NSS libraries for Windows

I'm creating this "idea" because I had opened a support case with a company called Deltek concerning one of their products that uses Cognos Analytics. Cognos does not properly report its capability to use higher strength TLS ciphers and signature algorithms when making SSL requests to other systems and because of this, some of these other systems which have high strength certificates reject the connection. The outcome of this case (response from IBM tech analyst TJ Nickerson below) was basically that IBM doesn't consider this issue a bug and I would have to create a feature request. So here we are.

First, some background.

We had a problem configuring a Cognos LDAP namespace running on Windows to communicate with an Active Directory server over SSL. At the time, we were struggling to figure out why Cognos refused to connect to our LDAP server over SSL after upgrading from version 11.0.6 to 11.0.13. The LDAP server simply rejects the TLS handshake and throws errors complaining about no matching cipher suites. We went back and forth with support, sending Wireshark traces but still could not figure out the issue. IBM support seemed to think that the ciphers were fine and that maybe the problem was in something else but it looks like we ended up giving up and moving on to other things.

Recently we ended up performing another upgrade to version 11.1.7, hoping that would fix the problem, but it did not so my boss asked me to take another look. After testing a number of things and researching a bit, I have found and verified the problem. Turns out the issue is partially a Windows issue and partially a Cognos issue but ultimately boils down to the signature algorithms that Cognos sends along with the TLS 1.2 handshake. Our LDAP server is running Windows Server 2012 R2 and the SSL certificate its using was created with the SHA512 signature algorithm. Cognos does not include that in its list of accepted algorithms when performing the handshake, even if the Cryptography config has the "Digest algorithm" property set to "SHA-512".

Now I state that its partially a Windows issue as well, because apparently Windows Server 2012 R2 is a little strict with respect to the TLSv1.2 specification. According to this document (https://docs.microsoft.com/en-us/windows-server/security/tls/tls-schannel-ssp-changes-in-windows-10-and-windows-server), Windows 2012 R2 strictly adheres to the TLSv1.2 spec which states that if the client provides a "signature_algorithms" extension in the TLS handshake, then all certificates provided by the server MUST be signed by a hash/signature algorithm pair that appears in that extension. The extensions reported by Cognos (running on Windows) during the SSL handshake are the following:

rsa_pkcs1_sha256

rsa_pkcs1_sha384

rsa_pkcs1_sha1

ecdsa_secp256r1_sha256

ecdsa_secp384r1_sha384

ecdsa_sha1

SHA256 DSA

SHA1 DSA

As you can see, no support of SHA512 is reported. Because of this, Windows 2012 R2 is rejecting the connection because its certificate is SHA512 and the client reports that it cannot accept that. Starting in Windows 10 (Windows Server 2016), apparently the constraints are relaxed and the server can send a certificate that does not comply with the client requested signature algorithm, if that's the server's only option.

I have verified that Cognos can make TLSv1.1 connections to our LDAP server by disabling TLSv1.2 on it. I also verified that Cognos can make TLSv1.2 connections to other LDAP servers running Windows Server 2016. Because both of these worked, I installed a new SSL certificate on our Windows Sever 2012 R2 machine using the SHA256 signature algorithm and Cognos can connect just fine.

So long story short, I'd like to have the Cognos legacy namespace provider(s) updated to allow for SHA512 signature algorithms. Apparently this will require new NSS libraries for Windows installs.

This NSS stuff is super old. Ideally it would be better to simply get rid of it and re-write the "legacy" providers in a way that is 100% Java based. I think some work has already been done in this respect because a relatively unknown "jldap" namespace is packaged in the product but cannot be configured properly.

Anyway - my "idea" (if you wanna call it that) is to either upgrade the old, crappy NSS libraries on Windows so that support for higher strength encryption works or get rid of that junk completely.

Here is the final response from IBM on my Deltek support case:

Tj.Nickerson (IBM)

Hello,

I have determined what is going on here.

On Windows, the NSS libraries that we ship with our product are older and as a result, the ciphers and algorithms that are used are limited. On Windows, only 8 signature algorithms are used:

Signature Hash Algorithms (8 algorithms)

Signature Algorithm: rsa_pkcs1_sha256 (0x0401)

Signature Algorithm: rsa_pkcs1_sha384 (0x0501)

Signature Algorithm: rsa_pkcs1_sha1 (0x0201)

Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)

Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)

Signature Algorithm: ecdsa_sha1 (0x0203)

Signature Algorithm: SHA256 DSA (0x0402)

Signature Algorithm: SHA1 DSA (0x0202)

This is not considered a defect and is the current product functionality on Windows. This would need to be treated as more of an enhancement request. It is certainly a valid enhancement request and we would consider supporting/bundling a newer version of NSS on Windows installs. I strongly recommend that you log an enhancement request for this on our Analytics Ideas page: https://ibm-data-and-ai.ideas.ibm.com/

Unfortunately upgrading NSS on Windows is not easy. For the time being, your best bet is to connect to your secured LDAP on Windows 2016 as that will get around the strict limitations enforced by Windows 2012.