Add documentation for lb-check

Closes #25077

Signed-off-by: Michal Hajas <mhajas@redhat.com>
Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
Co-authored-by: Alexander Schwartz <aschwart@redhat.com>

docs/documentation/release_notes/topics/24_0_0.adoc

@@ -6,8 +6,20 @@ The Keycloak JS adapter now uses the https://webpack.js.org/guides/package-expor
 
 Keycloak introduces an improved truststores configuration options. The Keycloak truststore is now used across the server: for outgoing connections, mTLS, database drivers and more. It's no longer needed to configure separate truststores for individual areas. To configure the truststore, you can put your truststores files or certificates in the default `conf/truststores`, or use the new `truststore-paths` config option. For details refer to the relevant https://www.keycloak.org/server/keycloak-truststore[guide].
 
-== Automatic certificate management for SAML identity providers
+= Automatic certificate management for SAML identity providers
 
 The SAML identity providers can now be configured to automatically download the signing certificates from the IDP entity metadata descriptor endpoint. In order to use the new feature the option `Metadata descriptor URL` should be configured in the provider (URL where the IDP metadata information with the certificates is published) and `Use metadata descriptor URL` needs to be `ON`. The certificates are automatically downloaded and cached in the `public-key-storage` SPI from that URL. The certificates can also be reloaded or imported from the admin console, using the action combo in the provider page.
 
 See the https://www.keycloak.org/docs/latest/server_admin/index.html#saml-v2-0-identity-providers[documentation] for more details about the new options.
+
+= Non-blocking health check for load balancers
+
+A new health check endpoint available at `/lb-check` was added.
+The execution is running in the event loop which means this check is responsive also in overloaded situations when Keycloak needs to handle many requests waiting in request queue.
+This behavior is useful, for example, in multi-site deployment where we do not want to fail over to the other site under heavy load.
+The endpoint is currently checking availability of the embedded and external Infinispan caches. Other checks may be added later.
+
+
+This endpoint is not available by default.
+To enable it, run Keycloak with feature `multi-site`.
+Proceed to https://www.keycloak.org/server/features[Enabling and disabling features] guide for more details.

docs/guides/high-availability/bblocks-active-passive-sync.adoc

@@ -57,9 +57,9 @@ A clustered deployment of {project_name} in each site, connected to an external
 
 </@tmpl.guide>
 
-== Loadbalancer
+== Load balancer
 
-A loadbalancer which checks the `/health/live` URl of the {project_name} deployment in each site.
+A load balancer which checks the `/lb-check` URl of the {project_name} deployment in each site.
 
 *Blueprint:* <@links.ha id="deploy-aws-route53-loadbalancer"/>.
 

docs/guides/high-availability/connect-keycloak-to-external-infinispan.adoc

@@ -69,11 +69,11 @@ include::examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn]
 <1> Custom cache configuration XML file definition, which includes configuration for remote or embedded {jdgserver_name} store.
 <2> The hostname and port of the remote cache {jdgserver_name} cluster.
 <3> The credentials required, username and password, to access the remote cache {jdgserver_name} cluster.
-<4> `jboss.site.name` is an arbitrary {jdgserver_name} site name which {project_name} needs for its embedded {jdgserver_name} deployment when a remote store is used.
+<4> Mounting the cache configuration Volume in Kubernetes.
-This site name is related only to the embedded {jdgserver_name} and does not need to match any value from the external {jdgserver_name} deployment.
-<5> Mounting the cache configuration Volume in Kubernetes.
 However, matching the `jboss.site.name` with the external {jdgserver_name} deployment site name helps debugging possible future issues.
 If you are using multiple sites for {project_name} in a cross-DC setup such as <@links.ha id="deploy-infinispan-kubernetes-crossdc" />, the site name must be different in each site.
+<5> `jboss.site.name` is an arbitrary {jdgserver_name} site name which {project_name} needs for its embedded {jdgserver_name} deployment when a remote store is used.
+This site name is related only to the embedded {jdgserver_name} and does not need to match any value from the external {jdgserver_name} deployment.
 <6> Defining the cache configuration Volume using the already created ConfigMap in Kubernetes.
 
 </@tmpl.guide>

docs/guides/high-availability/deploy-aws-route53-loadbalancer.adoc

@@ -90,6 +90,7 @@ aws elb describe-load-balancers \
     }
 ]
 ----
+<1> Utilise the Hosted Zone ID retrieved in the previous step
 +
 . Create Route53 health checks
 +
@@ -108,7 +109,7 @@ function createHealthCheck() {
   --health-check-config '
   {
     "Type": "HTTPS",
-    "ResourcePath": "/health/live",
+    "ResourcePath": "/lb-check",
     "FullyQualifiedDomainName": "'$1'",
     "Port": 443,
     "RequestInterval": 30,

docs/guides/high-availability/deploy-infinispan-kubernetes-crossdc.adoc

@@ -183,7 +183,6 @@ For `{site-b}`, the Cache CR is similar except in point 2.
 include::examples/generated/ispn-site-b.yaml[tag=infinispan-cache-sessions]
 ----
 
-[#verifying-the-deployment]
 == Verifying the deployment
 
 Confirm that the {jdgserver_name} cluster is formed, and the cross-site connection is established between the {ocp} clusters.

docs/guides/high-availability/deploy-keycloak-kubernetes.adoc

@@ -33,14 +33,15 @@ include::examples/generated/keycloak.yaml[tag=keycloak]
 Adjust this number to meet the needs of your system.
 As most requests will not touch the database due to the {project_name} embedded cache, this change can server several hundreds of requests per second.
 See the <@links.ha id="concepts-database-connections" /> {section} for details.
-<2> To be able to analyze the system under load, enable the metrics endpoint.
+<2> Enable additional features for multi-site support like the loadbalancer probe `/lb-check`.
+<3> To be able to analyze the system under load, enable the metrics endpoint.
 The disadvantage of the setting is that the metrics will be available at the external {project_name} endpoint, so you must add a filter so that the endpoint is not available from the outside.
 Use a reverse proxy in front of {project_name} to filter out those URLs.
-<3> The default setting for the internal JGroup thread pools is 200 threads maximum.
+<4> The default setting for the internal JGroup thread pools is 200 threads maximum.
 The number of all {project_name} threads in the StatefulSet should not exceed the number of JGroup threads to avoid a JGroup thread pool exhaustion which could stall {project_name} request processing.
 You might consider limiting the number of {project_name} threads further because multiple concurrent threads will lead to throttling by Kubernetes once the requested CPU limit is reached.
 See the <@links.ha id="concepts-threads" /> {section} for details.
-<4> The JVM options set additional parameters:
+<5> The JVM options set additional parameters:
 * `jgroups.thread_dumps_threshold` ensures that a log message "`thread pool is full`" appears once the JGroup thread pool is full for the first time.
 See the <@links.ha id="concepts-threads" /> {section} for details.
 * Adjust the memory settings for the heap.

docs/guides/high-availability/examples/generated/ispn-single.yaml

@@ -25,6 +25,17 @@ data:
           namesAsTags: true
           gauges: true
           histograms: false
+      server:
+        endpoints:
+          - securityRealm: default
+            socketBinding: default
+            connectors:
+              rest:
+                restConnector:
+                  authentication:
+                    mechanisms: BASIC
+              hotrod:
+                hotrodConnector: null
 ---
 # Source: ispn-helm/templates/infinispan.yaml
 # tag::infinispan-cache-actionTokens[]

docs/guides/high-availability/examples/generated/ispn-site-a.yaml

@@ -25,6 +25,17 @@ data:
           namesAsTags: true
           gauges: true
           histograms: false
+      server:
+        endpoints:
+          - securityRealm: default
+            socketBinding: default
+            connectors:
+              rest:
+                restConnector:
+                  authentication:
+                    mechanisms: BASIC
+              hotrod:
+                hotrodConnector: null
 ---
 # Source: ispn-helm/templates/infinispan.yaml
 # tag::infinispan-crossdc-status[]

docs/guides/high-availability/examples/generated/ispn-site-b.yaml

@@ -25,6 +25,17 @@ data:
           namesAsTags: true
           gauges: true
           histograms: false
+      server:
+        endpoints:
+          - securityRealm: default
+            socketBinding: default
+            connectors:
+              rest:
+                restConnector:
+                  authentication:
+                    mechanisms: BASIC
+              hotrod:
+                hotrodConnector: null
 ---
 # Source: ispn-helm/templates/infinispan.yaml
 # tag::infinispan-crossdc-status[]

docs/guides/high-availability/examples/generated/keycloak-ispn.yaml

@@ -129,7 +129,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/> <!--5-->
             </distributed-cache>
             <!--end::keycloak-ispn-remotestore[] -->
             <distributed-cache name="authenticationSessions" owners="2" statistics="true">
@@ -158,7 +157,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
             <distributed-cache name="offlineSessions" owners="2" statistics="true">
                 <expiration lifespan="-1"/>
@@ -186,7 +184,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
             <distributed-cache name="clientSessions" owners="2" statistics="true">
                 <expiration lifespan="-1"/>
@@ -213,7 +210,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
             <distributed-cache name="offlineClientSessions" owners="2" statistics="true">
                 <expiration lifespan="-1"/>
@@ -240,7 +236,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
             <distributed-cache name="loginFailures" owners="2" statistics="true">
                 <expiration lifespan="-1"/>
@@ -267,7 +262,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
             <local-cache name="authorization" simple-cache="true" statistics="true">
                 <encoding>
@@ -342,7 +336,6 @@ data:
                         </security>
                     </remote-store>
                 </persistence>
-                <state-transfer enabled="false"/>
             </distributed-cache>
         </cache-container>
     </infinispan>
@@ -713,10 +706,6 @@ spec:
             path: /
             pathType: ImplementationSpecific
 ---
-
-# Source: keycloak/templates/cockroach-operator/cockroach-operator.yaml
-# sourced from https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/install/operator.yaml
----
 # Source: keycloak/templates/keycloak.yaml
 # There are several callouts in this YAML marked with `# <1>' etc. See 'running/keycloak-deployment.adoc` for the details.
 # tag::keycloak[]
@@ -744,6 +733,9 @@ spec:
     passwordSecret:
       name: keycloak-db-secret
       key: password
+  features:
+    enabled:
+      - multi-site # <2>
   # tag::keycloak-ispn[]
   additionalOptions:
     - name: cache-config-file # <1>
@@ -752,7 +744,7 @@ spec:
       # end::keycloak-ispn[]
 
       value: json
-    - name: metrics-enabled # <2>
+    - name: metrics-enabled # <3>
       value: 'true'
     # tag::keycloak-ispn[]
     - name: remote-store-host # <2>
@@ -778,7 +770,7 @@ spec:
       # end::keycloak-ispn[]
       metadata:
         annotations:
-          checksum/config: c6c58fd5275c8f2be97e8dd2a535901ad5f3a6ad281890361c84e3f4b36c95f8-4832924b47210161956e3b1718daf07ff52d801545186a76c391485eaf1897d3--302cb302c7823761a9780d87abdb954ae8b71460dbf49779dbda7c66069b7938-v1.27.0
+          checksum/config: c6c58fd5275c8f2be97e8dd2a535901ad5f3a6ad281890361c84e3f4b36c95f8-4832924b47210161956e3b1718daf07ff52d801545186a76c391485eaf1897d3--56f92cd9012613402e2a7a61aded0f218d077b8c6345b22922ca7bf1a5c64984-v1.27.0
       # tag::keycloak[]
       # tag::keycloak-ispn[]
       spec:
@@ -788,7 +780,7 @@ spec:
         containers:
           - env:
               # end::keycloak-ispn[]
-              - name: 'QUARKUS_THREAD_POOL_MAX_THREADS' # <3>
+              - name: 'QUARKUS_THREAD_POOL_MAX_THREADS' # <4>
                 value: "200"
               # end::keycloak[]
               # tag::keycloak-queue-size[]
@@ -810,7 +802,7 @@ spec:
                     optional: false
                 # tag::keycloak[]
               # tag::keycloak-ispn[]
-              - name: JAVA_OPTS_APPEND # <4>
+              - name: JAVA_OPTS_APPEND # <5>
                 value: >
                   -Djgroups.thread_dumps_threshold=1 -Djboss.site.name=keycloak
 
@@ -834,7 +826,7 @@ spec:
             #       - 'true'
             # tag::keycloak-ispn[]
             volumeMounts:
-              - name: kcb-infinispan-cache-config # <5>
+              - name: kcb-infinispan-cache-config # <4>
                 mountPath: /opt/keycloak/conf/kcb-infinispan-cache-remote-store-config.xml
                 subPath: kcb-infinispan-cache-remote-store-config.xml
                 readOnly: true

docs/guides/high-availability/examples/generated/keycloak.yaml

@@ -400,10 +400,6 @@ spec:
             path: /
             pathType: ImplementationSpecific
 ---
-
-# Source: keycloak/templates/cockroach-operator/cockroach-operator.yaml
-# sourced from https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/install/operator.yaml
----
 # Source: keycloak/templates/keycloak.yaml
 # There are several callouts in this YAML marked with `# <1>' etc. See 'running/keycloak-deployment.adoc` for the details.
 # tag::keycloak[]
@@ -431,11 +427,14 @@ spec:
     passwordSecret:
       name: keycloak-db-secret
       key: password
+  features:
+    enabled:
+      - multi-site # <2>
   # tag::keycloak-ispn[]
   additionalOptions:
     - name: log-console-output
       value: json
-    - name: metrics-enabled # <2>
+    - name: metrics-enabled # <3>
       value: 'true'
   http:
     tlsSecret: keycloak-tls-secret
@@ -457,7 +456,7 @@ spec:
         containers:
           - env:
               # end::keycloak-ispn[]
-              - name: 'QUARKUS_THREAD_POOL_MAX_THREADS' # <3>
+              - name: 'QUARKUS_THREAD_POOL_MAX_THREADS' # <4>
                 value: "66"
               # end::keycloak[]
               # tag::keycloak-queue-size[]
@@ -479,7 +478,7 @@ spec:
                     optional: false
                 # tag::keycloak[]
               # tag::keycloak-ispn[]
-              - name: JAVA_OPTS_APPEND # <4>
+              - name: JAVA_OPTS_APPEND # <5>
                 value: >
                   -Xms512m -Xmx1524m -Djgroups.thread_dumps_threshold=1