Encrypt network communication in JGroups

Closes #25702 

Signed-off-by: Pedro Ruivo <pruivo@redhat.com>
Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
Co-authored-by: Alexander Schwartz <aschwart@redhat.com>

docs/guides/server/caching.adoc

@@ -236,7 +236,19 @@ By default, the value set to the `cache-stack` option has precedence over the tr
 If you are defining a custom stack, make sure the `cache-stack` option is not used for the custom changes to take effect.
 
 == Securing cache communication
-The current Infinispan cache implementation should be secured by various security measures such as RBAC, ACLs, and Transport stack encryption. For more information about securing cache communication, see the https://infinispan.org/docs/dev/titles/security/security.html#[Infinispan security guide].
+The current Infinispan cache implementation should be secured by various security measures such as RBAC, ACLs, and transport stack encryption.
+
+JGroups handles all the communication between {project_name} server, and it supports Java SSL sockets for TCP communication.
+{project_name} uses CLI options to configure the TLS communication without having to create a customized JGroups stack or modifying the cache XML file.
+
+To enable TLS, `cache-embedded-mtls-enabled` must be set to `true`.
+It requires a keystore with the certificate to use: `cache-embedded-mtls-key-store-file` sets the path to the keystore, and `cache-embedded-mtls-key-store-password` sets the password to decrypt it.
+The truststore contains the valid certificates to accept connection from, and it can be configured with `cache-embedded-mtls-trust-store-file` (path to the truststore), and `cache-embedded-mtls-trust-store-password` (password to decrypt it).
+To restrict unauthorized access, use a self-signed certificate for each {project_name} deployment.
+
+For JGroups stacks with `UDP` or `TCP_NIO2`, see the http://jgroups.org/manual5/index.html#ENCRYPT[JGroups Encryption documentation] on how to set up the protocol stack.
+
+For more information about securing cache communication, see the https://infinispan.org/docs/stable/titles/security/security.html#[Infinispan security guide].
 
 == Exposing metrics from caches
 

docs/guides/server/configuration-production.adoc

@@ -16,6 +16,8 @@ This {section} describes the general areas of configuration required for a produ
 
 To configure secure communication channels for {project_name}, see <@links.server id="enabletls"/> and <@links.server id="outgoinghttp"/>.
 
+To secure the cache communication for {project_name}, see <@links.server id="caching"/>.
+
 == The hostname for {project_name}
 In a production environment, {project_name} instances usually run in a private network, but {project_name} needs to expose certain public facing endpoints to communicate with the applications to be secured.
 

quarkus/config-api/src/main/java/org/keycloak/config/CachingOptions.java

@@ -4,6 +4,13 @@ import java.io.File;
 
 public class CachingOptions {
 
+    private static final String CACHE_EMBEDDED_MTLS_PREFIX = "cache-embedded-mtls";
+    public static final String CACHE_EMBEDDED_MTLS_ENABLED_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-enabled";
+    public static final String CACHE_EMBEDDED_MTLS_KEYSTORE_FILE_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-key-store-file";
+    public static final String CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-key-store-password";
+    public static final String CACHE_EMBEDDED_MTLS_TRUSTSTORE_FILE_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-trust-store-file";
+    public static final String CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD_PROPERTY = CACHE_EMBEDDED_MTLS_PREFIX + "-trust-store-password";
+
     public enum Mechanism {
         ispn,
         local
@@ -40,4 +47,39 @@ public class CachingOptions {
                     + "The configuration file is relative to the 'conf/' directory.")
             .buildTime(true)
             .build();
+
+    public static final Option<Boolean> CACHE_EMBEDDED_MTLS_ENABLED = new OptionBuilder<>(CACHE_EMBEDDED_MTLS_ENABLED_PROPERTY, Boolean.class)
+            .category(OptionCategory.CACHE)
+            .description("Encrypts the network communication between Keycloak servers.")
+            .defaultValue(Boolean.FALSE)
+            .buildTime(true)
+            .build();
+
+    public static final Option<String> CACHE_EMBEDDED_MTLS_KEYSTORE = new OptionBuilder<>(CACHE_EMBEDDED_MTLS_KEYSTORE_FILE_PROPERTY, String.class)
+            .category(OptionCategory.CACHE)
+            .description("The Keystore file path. The Keystore must contain the certificate to use by the TLS protocol. " +
+                    "By default, it lookup 'cache-mtls-keystore.p12' under conf/ directory.")
+            .buildTime(true)
+            .build();
+
+    public static final Option<String> CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD = new OptionBuilder<>(CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD_PROPERTY, String.class)
+            .category(OptionCategory.CACHE)
+            .description("The password to access the Keystore.")
+            .buildTime(true)
+            .build();
+
+    public static final Option<String> CACHE_EMBEDDED_MTLS_TRUSTSTORE = new OptionBuilder<>(CACHE_EMBEDDED_MTLS_TRUSTSTORE_FILE_PROPERTY, String.class)
+            .category(OptionCategory.CACHE)
+            .description("The Truststore file path. " +
+                    "It should contain the trusted certificates or the Certificate Authority that signed the certificates. " +
+                    "By default, it lookup 'cache-mtls-truststore.p12' under conf/ directory.")
+            .buildTime(true)
+            .build();
+
+    public static final Option<String> CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD = new OptionBuilder<>(CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD_PROPERTY, String.class)
+            .category(OptionCategory.CACHE)
+            .description("The password to access the Truststore.")
+            .buildTime(true)
+            .build();
+
 }

quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/configuration/mappers/CachingPropertyMappers.java

@@ -9,6 +9,7 @@ import static java.util.Optional.of;
 import static org.keycloak.quarkus.runtime.configuration.mappers.PropertyMapper.fromOption;
 
 import java.io.File;
+import java.nio.file.Paths;
 import java.util.Optional;
 
 final class CachingPropertyMappers {
@@ -30,7 +31,23 @@ final class CachingPropertyMappers {
                         .to("kc.spi-connections-infinispan-quarkus-config-file")
                         .transformer(CachingPropertyMappers::resolveConfigFile)
                         .paramLabel("file")
-                        .build()
+                        .build(),
+                fromOption(CachingOptions.CACHE_EMBEDDED_MTLS_ENABLED)
+                        .build(),
+                fromOption(CachingOptions.CACHE_EMBEDDED_MTLS_KEYSTORE.withRuntimeSpecificDefault(getDefaultKeystorePathValue()))
+                        .paramLabel("file")
+                        .build(),
+                fromOption(CachingOptions.CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD)
+                        .paramLabel("password")
+                        .isMasked(true)
+                        .build(),
+                fromOption(CachingOptions.CACHE_EMBEDDED_MTLS_TRUSTSTORE.withRuntimeSpecificDefault(getDefaultTruststorePathValue()))
+                        .paramLabel("file")
+                        .build(),
+                fromOption(CachingOptions.CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD)
+                        .paramLabel("password")
+                        .isMasked(true)
+                        .build(),
         };
     }
 
@@ -52,4 +69,32 @@ final class CachingPropertyMappers {
 
         return of(pathPrefix + value.get());
     }
+
+    private static String getDefaultKeystorePathValue() {
+        String homeDir = Environment.getHomeDir();
+
+        if (homeDir != null) {
+            File file = Paths.get(homeDir, "conf", "cache-mtls-keystore.p12").toFile();
+
+            if (file.exists()) {
+                return file.getAbsolutePath();
+            }
+        }
+
+        return null;
+    }
+
+    private static String getDefaultTruststorePathValue() {
+        String homeDir = Environment.getHomeDir();
+
+        if (homeDir != null) {
+            File file = Paths.get(homeDir, "conf", "cache-mtls-truststore.p12").toFile();
+
+            if (file.exists()) {
+                return file.getAbsolutePath();
+            }
+        }
+
+        return null;
+    }
 }

quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/storage/legacy/infinispan/CacheManagerFactory.java

@@ -20,22 +20,35 @@ package org.keycloak.quarkus.runtime.storage.legacy.infinispan;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
-import java.util.concurrent.ThreadFactory;
 import java.util.concurrent.TimeUnit;
 
 import io.micrometer.core.instrument.Metrics;
+import org.infinispan.configuration.global.GlobalConfiguration;
 import org.infinispan.configuration.parsing.ConfigurationBuilderHolder;
 import org.infinispan.configuration.parsing.ParserRegistry;
 import org.infinispan.jboss.marshalling.core.JBossUserMarshaller;
 import org.infinispan.manager.DefaultCacheManager;
 import org.infinispan.metrics.config.MicrometerMeterRegisterConfigurationBuilder;
+import org.infinispan.remoting.transport.jgroups.JGroupsTransport;
 import org.jboss.logging.Logger;
+import org.jgroups.protocols.TCP_NIO2;
+import org.jgroups.protocols.UDP;
+import org.jgroups.util.TLS;
+import org.jgroups.util.TLSClientAuth;
 import org.keycloak.quarkus.runtime.configuration.Configuration;
 
+import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_MTLS_ENABLED_PROPERTY;
+import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_MTLS_KEYSTORE_FILE_PROPERTY;
+import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD_PROPERTY;
+import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_MTLS_TRUSTSTORE_FILE_PROPERTY;
+import static org.keycloak.config.CachingOptions.CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD_PROPERTY;
+
 public class CacheManagerFactory {
 
+    private static final Logger logger = Logger.getLogger(CacheManagerFactory.class);
+
     private String config;
-    private boolean metricsEnabled;
+    private final boolean metricsEnabled;
     private DefaultCacheManager cacheManager;
     private Future<DefaultCacheManager> cacheManagerFuture;
     private ExecutorService executor;
@@ -68,12 +81,7 @@ public class CacheManagerFactory {
     }
 
     private ExecutorService createThreadPool() {
-        return Executors.newSingleThreadExecutor(new ThreadFactory() {
+        return Executors.newSingleThreadExecutor(r -> new Thread(r, "keycloak-cache-init"));
-            @Override
-            public Thread newThread(Runnable r) {
-                return new Thread(r, "keycloak-cache-init");
-            }
-        });
     }
 
     private DefaultCacheManager startCacheManager() {
@@ -129,8 +137,51 @@ public class CacheManagerFactory {
     private void configureTransportStack(ConfigurationBuilderHolder builder) {
         String transportStack = Configuration.getRawValue("kc.cache-stack");
 
+        var transportConfig = builder.getGlobalConfigurationBuilder().transport();
         if (transportStack != null && !transportStack.isBlank()) {
-            builder.getGlobalConfigurationBuilder().transport().defaultTransport().stack(transportStack);
+            transportConfig.defaultTransport().stack(transportStack);
         }
+
+        if (booleanProperty(CACHE_EMBEDDED_MTLS_ENABLED_PROPERTY)) {
+            validateTlsAvailable(transportConfig.build());
+            var tls = new TLS()
+                    .enabled(true)
+                    .setKeystorePath(requiredStringProperty(CACHE_EMBEDDED_MTLS_KEYSTORE_FILE_PROPERTY))
+                    .setKeystorePassword(requiredStringProperty(CACHE_EMBEDDED_MTLS_KEYSTORE_PASSWORD_PROPERTY))
+                    .setKeystoreType("pkcs12")
+                    .setTruststorePath(requiredStringProperty(CACHE_EMBEDDED_MTLS_TRUSTSTORE_FILE_PROPERTY))
+                    .setTruststorePassword(requiredStringProperty(CACHE_EMBEDDED_MTLS_TRUSTSTORE_PASSWORD_PROPERTY))
+                    .setTruststoreType("pkcs12")
+                    .setClientAuth(TLSClientAuth.NEED)
+                    .setProtocols(new String[]{"TLSv1.3"});
+            transportConfig.addProperty(JGroupsTransport.SOCKET_FACTORY, tls.createSocketFactory());
+            Logger.getLogger(CacheManagerFactory.class).info("MTLS enabled for communications for embedded caches");
+        }
+    }
+
+    private void validateTlsAvailable(GlobalConfiguration config) {
+        var stackName = config.transport().stack();
+        if (stackName == null) {
+            // unable to validate
+            return;
+        }
+        for (var protocol : config.transport().jgroups().configurator(stackName).getProtocolStack()) {
+            var name = protocol.getProtocolName();
+            if (name.equals(UDP.class.getSimpleName()) ||
+                    name.equals(UDP.class.getName()) ||
+                    name.equals(TCP_NIO2.class.getSimpleName()) ||
+                    name.equals(TCP_NIO2.class.getName())) {
+                throw new RuntimeException("Cache TLS is not available with protocol " + name);
+            }
+        }
+
+    }
+
+    private static boolean booleanProperty(String propertyName) {
+        return Configuration.getOptionalKcValue(propertyName).map(Boolean::parseBoolean).orElse(Boolean.FALSE);
+    }
+
+    private static String requiredStringProperty(String propertyName) {
+        return Configuration.getOptionalKcValue(propertyName).orElseThrow(() -> new RuntimeException("Property " + propertyName + " required but not specified"));
     }
 }

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testBuildHelp.unix.approved.txt

@@ -27,6 +27,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testBuildHelp.windows.approved.txt

@@ -27,6 +27,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelp.unix.approved.txt

@@ -25,6 +25,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelp.windows.approved.txt

@@ -25,6 +25,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.
@@ -76,11 +90,11 @@ Feature:
                        account2[:v1], account3[:v1], admin-api[:v1], admin-fine-grained-authz[:v1],
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
-                       dynamic-scopes[:v1], fips[:v1], hostname[:v1], impersonation[:v1], js-adapter
+                       dynamic-scopes[:v1], fips[:v1], hostname[:v1],impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], kerberos[:v1], linkedin-oauth[:v1], multi-site[:v1],
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
-                       offline-session-preloading[:v1], par[:v1], preview, recovery-codes[:v1],
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
-                       scripts[:v1], step-up-authentication[:v1], token-exchange[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
-                       transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelpAll.unix.approved.txt

@@ -25,6 +25,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelpAll.windows.approved.txt

@@ -25,6 +25,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.
@@ -76,11 +90,11 @@ Feature:
                        account2[:v1], account3[:v1], admin-api[:v1], admin-fine-grained-authz[:v1],
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
-                       dynamic-scopes[:v1], fips[:v1], hostname[:v1], impersonation[:v1], js-adapter
+                       dynamic-scopes[:v1], fips[:v1], hostname[:v1],impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], kerberos[:v1], linkedin-oauth[:v1], multi-site[:v1],
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
-                       offline-session-preloading[:v1], par[:v1], preview, recovery-codes[:v1],
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
-                       scripts[:v1], step-up-authentication[:v1], token-exchange[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
-                       transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartHelp.unix.approved.txt

@@ -26,6 +26,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartHelp.windows.approved.txt

@@ -26,6 +26,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.
@@ -77,11 +91,11 @@ Feature:
                        account2[:v1], account3[:v1], admin-api[:v1], admin-fine-grained-authz[:v1],
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
-                       dynamic-scopes[:v1], fips[:v1], hostname[:v1], impersonation[:v1], js-adapter
+                       dynamic-scopes[:v1], fips[:v1], hostname[:v1],impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], kerberos[:v1], linkedin-oauth[:v1], multi-site[:v1],
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
-                       offline-session-preloading[:v1], par[:v1], preview, recovery-codes[:v1],
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
-                       scripts[:v1], step-up-authentication[:v1], token-exchange[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
-                       transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartHelpAll.unix.approved.txt

@@ -26,6 +26,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartHelpAll.windows.approved.txt

@@ -26,6 +26,20 @@ Cache:
 --cache-config-file <file>
                      Defines the file from which cache configuration should be loaded from. The
                        configuration file is relative to the 'conf/' directory.
+--cache-embedded-mtls-enabled <true|false>
+                     Encrypts the network communication between Keycloak servers. Default: false.
+--cache-embedded-mtls-key-store-file <file>
+                     The Keystore file path. The Keystore must contain the certificate to use by
+                       the TLS protocol. By default, it lookup 'cache-mtls-keystore.p12' under
+                       conf/ directory.
+--cache-embedded-mtls-key-store-password <password>
+                     The password to access the Keystore.
+--cache-embedded-mtls-trust-store-file <file>
+                     The Truststore file path. It should contain the trusted certificates or the
+                       Certificate Authority that signed the certificates. By default, it lookup
+                       'cache-mtls-truststore.p12' under conf/ directory.
+--cache-embedded-mtls-trust-store-password <password>
+                     The password to access the Truststore.
 --cache-stack <stack>
                      Define the default stack to use for cluster communication and node discovery.
                        This option only takes effect if 'cache' is set to 'ispn'. Default: udp.
@@ -77,11 +91,11 @@ Feature:
                        account2[:v1], account3[:v1], admin-api[:v1], admin-fine-grained-authz[:v1],
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
-                       dynamic-scopes[:v1], fips[:v1], hostname[:v1], impersonation[:v1], js-adapter
+                       dynamic-scopes[:v1], fips[:v1], hostname[:v1],impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], kerberos[:v1], linkedin-oauth[:v1], multi-site[:v1],
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
-                       offline-session-preloading[:v1], par[:v1], preview, recovery-codes[:v1],
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
-                       scripts[:v1], step-up-authentication[:v1], token-exchange[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
-                       transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,