diff --git a/docs/documentation/upgrading/topics/changes/changes-26_1_0.adoc b/docs/documentation/upgrading/topics/changes/changes-26_1_0.adoc index db31b2e352..658a1ebfd0 100644 --- a/docs/documentation/upgrading/topics/changes/changes-26_1_0.adoc +++ b/docs/documentation/upgrading/topics/changes/changes-26_1_0.adoc @@ -22,7 +22,7 @@ These APIs are no longer needed as initialization is done automatically on deman = Virtual Threads enabled for Infinispan and JGroups thread pools Starting from this release, {project_name} automatically enables the virtual thread pool support in both the embedded Infinispan and JGroups when running on OpenJDK 21. -This removes the need to configure the thread pool and reduces overall memory footprint. +This removes the need to configure the JGroups thread pool, the need to align the JGroups thread pool with the HTTP worker thread pool, and reduces the overall memory footprint. To disable the virtual threads, add one of the Java system properties combinations to your deployment: * `-Dorg.infinispan.threads.virtual=false`: disables virtual thread in both Infinispan and JGroups. diff --git a/docs/guides/high-availability/concepts-threads.adoc b/docs/guides/high-availability/concepts-threads.adoc index e1dae9947e..4ff442cfd6 100644 --- a/docs/guides/high-availability/concepts-threads.adoc +++ b/docs/guides/high-availability/concepts-threads.adoc @@ -12,6 +12,15 @@ For a configuration where this is applied, visit <@links.ha id="deploy-keycloak- == Concepts +=== JGroups communications + +// remove this paragraph once OpenJDK 17 is no longer supported on the server side. +// https://github.com/keycloak/keycloak/issues/31101 + +JGroups communications, which is used in single-site setups for the communication between {project_name} nodes, benefits from the use of virtual threads which are available in OpenJDK 21. +This reduces the memory usage and removes the need to configure thread pool sizes. +Therefore, the use of OpenJDK 21 is recommended. + === Quarkus executor pool {project_name} requests, as well as blocking probes, are handled by an executor pool. Depending on the available CPU cores, it has a maximum size of 50 or more threads. @@ -31,32 +40,6 @@ If you increase the number of database connections and the number of threads too The number of database connections is configured via the link:{links_server_all-config_url}?q=db-pool[`Database` settings `db-pool-initial-size`, `db-pool-min-size` and `db-pool-max-size`] respectively. Low numbers ensure fast response times for all clients, even if there is an occasionally failing request when there is a load spike. -=== JGroups connection pool - -[NOTE] -==== -* This currently applies to single-site setups only. -In a multi-site setup with an external {jdgserver_name} this is not a restriction. -* This currently applies if virtual threads are disabled. -Since {project_name} 26.1, virtual threads are enabled in both embedded Infinispan and JGroups if running on OpenJDK 21 or higher. -==== - - -The combined number of executor threads in all {project_name} nodes in the cluster should not exceed too much the number of threads available in JGroups thread pool to avoid the warning `thread pool is full (max=, active=)`. - -The warning includes a thread dump when the Java system property `-Djgroups.thread_dumps_enabled=true` is set. -It may incur in a penalty in performance collecting those thread dumps. - --- -include::partials/threads/executor-jgroups-thread-calculation.adoc[] --- - -Use metrics to monitor the total JGroup threads in the pool and for the threads active in the pool. -When using TCP as the JGroups transport protocol, the metrics `vendor_jgroups_tcp_get_thread_pool_size` and `vendor_jgroups_tcp_get_thread_pool_size_active` are available for monitoring. When using UDP, the metrics `vendor_jgroups_udp_get_thread_pool_size` and `vendor_jgroups_udp_get_thread_pool_size_active` are available. -This is useful to monitor that limiting the Quarkus thread pool size keeps the number of active JGroup threads below the maximum JGroup thread pool size. - -WARNING: The metrics are not available when virtual threads are enabled in JGroups. - [#load-shedding] === Load Shedding diff --git a/docs/guides/high-availability/deploy-keycloak-kubernetes.adoc b/docs/guides/high-availability/deploy-keycloak-kubernetes.adoc index f66b9c89e1..acae4135ea 100644 --- a/docs/guides/high-availability/deploy-keycloak-kubernetes.adoc +++ b/docs/guides/high-availability/deploy-keycloak-kubernetes.adoc @@ -17,6 +17,7 @@ Use it together with the other building blocks outlined in the <@links.ha id="bb * Understanding of a <@links.operator id="basic-deployment" /> of {project_name} with the {project_name} Operator. * Aurora AWS database deployed using the <@links.ha id="deploy-aurora-multi-az" /> {section}. * {jdgserver_name} server deployed using the <@links.ha id="deploy-infinispan-kubernetes-crossdc" /> {section}. +* Running {project_name} with OpenJDK 21, which is the default for the containers distributed for {project_name}, as this enabled virtual threads for the JGroups communication. == Procedure @@ -46,8 +47,6 @@ See the <@links.ha id="concepts-database-connections" /> {section} for details. <5> To be able to analyze the system under load, enable the metrics endpoint. The disadvantage of the setting is that the metrics will be available at the external {project_name} endpoint, so you must add a filter so that the endpoint is not available from the outside. Use a reverse proxy in front of {project_name} to filter out those URLs. -<6> You might consider limiting the number of {project_name} threads further because multiple concurrent threads will lead to throttling by Kubernetes once the requested CPU limit is reached. -See the <@links.ha id="concepts-threads" /> {section} for details. == Verifying the deployment @@ -70,7 +69,11 @@ spec: additionalOptions: include::examples/generated/keycloak.yaml[tag=keycloak-queue-size] ---- + All exceeding requests are served with an HTTP 503. + +You might consider limiting the value for `http-pool-max-threads` further because multiple concurrent threads will lead to throttling by Kubernetes once the requested CPU limit is reached. + See the <@links.ha id="concepts-threads" /> {section} about load shedding for details. == Optional: Disable sticky sessions diff --git a/docs/guides/high-availability/examples/generated/keycloak-ispn.yaml b/docs/guides/high-availability/examples/generated/keycloak-ispn.yaml index 6adc4824e6..4dffd88d25 100644 --- a/docs/guides/high-availability/examples/generated/keycloak-ispn.yaml +++ b/docs/guides/high-availability/examples/generated/keycloak-ispn.yaml @@ -464,16 +464,16 @@ spec: - name: http-metrics-slos value: '5,10,25,50,250,500' # tag::keycloak[] + # end::keycloak[] # tag::keycloak-queue-size[] - name: http-max-queued-requests value: "1000" # end::keycloak-queue-size[] + # tag::keycloak[] - name: log-console-output value: json - name: metrics-enabled # <5> value: 'true' - - name: http-pool-max-threads # <6> - value: "200" # tag::keycloak-ispn[] - name: cache-remote-host # <1> value: "infinispan.keycloak.svc" diff --git a/docs/guides/high-availability/examples/generated/keycloak.yaml b/docs/guides/high-availability/examples/generated/keycloak.yaml index 8dade83c86..6680e6e927 100644 --- a/docs/guides/high-availability/examples/generated/keycloak.yaml +++ b/docs/guides/high-availability/examples/generated/keycloak.yaml @@ -453,16 +453,16 @@ spec: - name: http-metrics-slos value: '5,10,25,50,250,500' # tag::keycloak[] + # end::keycloak[] # tag::keycloak-queue-size[] - name: http-max-queued-requests value: "1000" # end::keycloak-queue-size[] + # tag::keycloak[] - name: log-console-output value: json - name: metrics-enabled # <5> value: 'true' - - name: http-pool-max-threads # <6> - value: "66" # end::keycloak[] # This block is just for documentation purposes as we need both versions of Infinispan config, with and without numbers to corresponding options # tag::keycloak[] @@ -510,6 +510,7 @@ spec: - name: JAVA_OPTS_APPEND # <5> value: "" ports: + # end::keycloak[] # readinessProbe: # exec: # command: diff --git a/docs/guides/high-availability/partials/threads/executor-jgroups-thread-calculation.adoc b/docs/guides/high-availability/partials/threads/executor-jgroups-thread-calculation.adoc deleted file mode 100644 index 9d007189ec..0000000000 --- a/docs/guides/high-availability/partials/threads/executor-jgroups-thread-calculation.adoc +++ /dev/null @@ -1,5 +0,0 @@ -The number of JGroup threads is `200` by default. -While it can be configured using the property Java system property `jgroups.thread_pool.max_threads`, we advise keeping it at this value. -As shown in experiments, the total number of Quarkus worker threads in the cluster should not exceed the number of threads in the JGroup thread pool of `200` in each node to avoid requests being dropped in the JGroups communication. -Given a {project_name} cluster with four nodes, each node should then have around 50 Quarkus worker threads. -Use the {project_name} configuration option `http-pool-max-threads` to configure the maximum number of Quarkus worker threads.