Service Registry & SSO Integration with Self-Signed Certificate

When securing Service Registry using Red Hat SSO that has HTTPS endpoint set up with a self-signed certificate, you might end up with following error when starting the Service Registry container.

__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2022-08-05 09:24:44,968 INFO  [org.apa.kaf.con.jso.JsonConverterConfig] (main) JsonConverterConfig values:
 converter.type = key
 decimal.format = BASE64
 schemas.cache.size = 0
 schemas.enable = true

Aug 05, 2022 9:24:47 AM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 3.2.0.Final-redhat-00003
Aug 05, 2022 9:24:48 AM io.quarkus.runtime.ApplicationLifecycleManager run
ERROR: Failed to start application (with profile prod)
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
 at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
 at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
 at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
 at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
 at java.base/sun.security.validator.Validator.validate(Validator.java:264)
 at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
 at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
 at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
 at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
 at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
 at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
 at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
 at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
 at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
 at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
 at java.base/java.security.AccessController.doPrivileged(Native Method)
 at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
 at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1557)
 at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1571)
 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1455)
 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282)
 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329)
 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.base/java.lang.Thread.run(Thread.java:829)

Problem

The reason behind the scence is basically Service Registry cannot connect to Red Hat SSO via HTTPS endpoint because self-signed certificate is unknown and the CA who sign the certificate is not trusted by JVM.

Solution

To fix this issue, in summary, you need to add the self-signed certificate (the one that's used to setup HTTPS endpoint for SSO) or the certificate of CA who signed the certificate itself to JVM truststore/keystore file so the certificate is known or trusted by JVM. To do so, follow the steps below:

  1. Get list of Service Registry pod(s) with follwing command.

    Sample output:

  2. SSH to Service Registry pod using oc rsh command and find JVM truststore/keystore path in the container with ls -l $JAVA_HOME/lib/security/ command. Look for the cacerts file which is the default JVM truststore/keystore file.

    However, in this case, the cacerts file is a symbolic link to /etc/pki/java/cacerts file which is another symbolic link to /etc/pki/ca-trust/extracted/java/cacerts file which is the file that we need to copy from container.

    Sample output:

  3. Use oc rsync command to copy file from Servie Registry container.

    Sample output:

  4. Add self-signed certificate to the cacerts file with following command line or GUI tool.

    • Replace $JDK_HOME with your actual JDK home path.

    • Replace $CERT with the path to your certificate the you previously installed to the system.

    • Replace $ALIAS with the preferred alias to be used in the keystore.

    • Note that changeit is the default password for Java's cacerts file.

  5. Create a Secret object for the cacerts truststore file.

    Create a Secret object
    Create a Secret object

    Note. If you encounter a java.io.IOException: Invalid keystore format error in Service Registry container log, the upload of the binary file did not work properly. As an alternative, encode the file as a base64 string using cat registry-keystore.jks | base64 -w0 > data.txt and edit the Secret resource as yaml to manually add the encoded file.

  6. Add the Secret to Service Registry Deployment.

    Add the Secret object to Service Registry Deployment

  7. You'll be directed to the Service Registry Deployment page.

    Service Registry Deployment

  8. Go to Environment tab and add two new environment variables as screenshot below.

    • TRUSTSTORE_PASSWORD environment variable value is read from the Secret object you've created above.

    • JAVA_OPTIONS environment value is -Djavax.net.ssl.trustStore=/etc/pki/java/custom/truststore -Djavax.net.ssl.trustStorePassword=$(TRUSTSTORE_PASSWORD)

    Add TRUSTSTORE_PASSWORD environment variable
    Add JAVA_OPTIONS environment variable

  9. After new pods are created, open one of the and go to Terminal tab then type ls -l /etc/pki/java/custom/ command. You should see the truststore file which is mounted from the Secret object.

    Truststore file is mounted to container

  10. Go to Logs tab, you should see the value of JAVA_OPTIONS environment variable get added as the java command arguments. And the application should start without any error about certificate.

    JAVA_OPTIONS is added as one of java command arguments

References

PreviousInstallation Helper Script

Last updated