summary="Understand these concepts to avoid resource exhaustion and congestion"
tileVisible="false" >
This section is intended when you want to understand the considerations and best practices on how to configure thread pools connection pools for {project_name}.
For a configuration where this is applied, visit <@links.ha id="deploy-keycloak-kubernetes" />.
{project_name} requests, as well as blocking probes, are handled by an executor pool. Depending on the available CPU cores, it has a maximum size of 50 or more threads.
{project_name} allows configuring the maximum thread pool size by the link:{links_server_all-config_url}?q=http-pool-max-threads[`http-pool-max-threads`] configuration option. See <@links.ha id="deploy-keycloak-kubernetes" /> for an example.
When running on Kubernetes, adjust the number of worker threads to avoid creating more load than what the CPU limit allows for the Pod to avoid throttling, which would lead to congestion.
If you increase the number of database connections and the number of threads too much, the system will be congested under a high load with requests queueing up, which leads to a bad performance.
The number of database connections is configured via the link:{links_server_all-config_url}?q=db-pool[`Database` settings `db-pool-initial-size`, `db-pool-min-size` and `db-pool-max-size`] respectively.
The combined number of executor threads in all {project_name} nodes in the cluster should not exceed too much the number of threads available in JGroups thread pool to avoid the warning `thread pool is full (max=<value>, active=<value>)`.
The warning includes a thread dump when the Java system property `-Djgroups.thread_dumps_enabled=true` is set.
It may incur in a penalty in performance collecting those thread dumps.
Use metrics to monitor the total JGroup threads in the pool and for the threads active in the pool.
When using TCP as the JGroups transport protocol, the metrics `vendor_jgroups_tcp_get_thread_pool_size` and `vendor_jgroups_tcp_get_thread_pool_size_active` are available for monitoring. When using UDP, the metrics `vendor_jgroups_udp_get_thread_pool_size` and `vendor_jgroups_udp_get_thread_pool_size_active` are available.
This is useful to monitor that limiting the Quarkus thread pool size keeps the number of active JGroup threads below the maximum JGroup thread pool size.
By default, {project_name} will queue all incoming requests infinitely, even if the request processing stalls.
This will use additional memory in the Pod, can exhaust resources in the load balancers, and the requests will eventually time out on the client side without the client knowing if the request has been processed.
To limit the number of queued requests in {project_name}, set an additional Quarkus configuration option.
{project_name}'s liveness probe is non-blocking to avoid a restart of a Pod under a high load.
// Developer's note: See KeycloakReadyHealthCheck for the details of the blocking/non-blocking behavior
The overall health probe and the readiness can probe in some cases block to check the connection to the database, so they might fail under a high load.
Due to this, a Pod can become non-ready under a high load.
In order for Java to create threads, when running on Linux it needs to have file handles available.
Therefore, the number of open files (as retrieved as `ulimit -n` on Linux) need to provide head-space for {project_name} to increase the number of threads needed.
Each thread will also consume memory, and the container memory limits need to be set to a value that allows for this or the Pod will be killed by Kubernetes.
