KEYCLOAK-5897 Support for multiple provisioning options in performance testsuite

testsuite/performance/README.docker-compose.md

@@ -1,35 +1,20 @@
-# Keycloak Performance Testsuite - Docker Compose
+# Keycloak Performance Testsuite - Docker Compose Provisioner
 
-## Requirements:
+## Supported Deployments
-- Maven 3.1.1+
-- Unpacked Keycloak server distribution in `keycloak/target` folder.
-- Docker 1.13+
-- Docker Compose 1.14+
 
-### Keycloak Server Distribution
+| Deployment   | Available Operations                                  | Orchestration Template          |
-To unpack the current Keycloak server distribution into `keycloak/target` folder:
+| ------------ | ----------------------------------------------------- | ------------------------------- |
-1. Build and install the distribution by running `mvn install -Pdistribution` from the root of the Keycloak project.
+| `singlenode` | `provision`, `teardown`, `export-dump`, `import-dump` | `docker-compose.yml`            |
-2. Unpack the installed artifact by running `mvn process-resources` from the Performance Testsuite module.
+| `cluster`    | `provision`, `teardown`, `export-dump`, `import-dump` | `docker-compose-cluster.yml`*   |
+| `crossdc`    | `provision`, `teardown`, `export-dump`, `import-dump` | `docker-compose-crossdc.yml`*   |
+| `monitoring` | `provision`, `teardown`                               | `docker-compose-monitoring.yml` |
 
-## Deployments
+The docker-compose orchestration templates are located in `tests/src/main/docker-compose` directory.
 
-### Singlenode Deployment
+**[*]** The cluster and crossdc templates are generated dynamically during the `provision` operation based on provided `cpusets` parameter.
-- Build / rebuild: `docker-compose build`
+One Keycloak service entry is generated for each cpuset. This is a workaround for limitations of the default docker-compose scaling mechanism 
-- Start services: `docker-compose up -d --build`
+which only allows setting `cpuset` per service, not per container. For more predictable performance results it is necessary for each 
-  Note: The `--build` parameter triggers a rebuild/restart of changed services if they are already running.
+Keycloak server to have an exclusive access to specific CPU cores.
-- Stop services: `docker-compose down -v`. If you wish to keep the container volumes skip the `-v` option.
-
-### Keycloak Cluster Deployment
-- Build / rebuild: `docker-compose -f docker-compose-cluster.yml build`
-- Start services: `docker-compose -f docker-compose-cluster.yml up -d --build`
-- Scaling KC nodes: `docker-compose -f docker-compose-cluster.yml up -d --build --scale keycloak=2`
-- Stop services: `docker-compose -f docker-compose-cluster.yml down -v`. If you wish to keep the container volumes skip the `-v` option.
-
-### Cross-DC Deployment
-- Build / rebuild: `docker-compose -f docker-compose-crossdc.yml build`
-- Start services: `docker-compose -f docker-compose-crossdc.yml up -d --build`
-- Scaling KC nodes: `docker-compose -f docker-compose-crossdc.yml up -d --build --scale keycloak_dc1=2 --scale keycloak_dc2=3`
-- Stop services: `docker-compose -f docker-compose-crossdc.yml down -v`. If you wish to keep the container volumes skip the `-v` option.
 
 ## Debugging docker containers:
 - List started containers: `docker ps`. It's useful to watch continuously: `watch docker ps`.

testsuite/performance/README.log-tool.md

@@ -18,8 +18,8 @@ LOG_DIR=$HOME/devel/keycloak/keycloak/testsuite/performance/tests/target/gatling
 
 Get general statistics about the run to help with deciding about the interval to extract:
 ```
-./log-tool.sh -s -f $LOG_DIR/simulation.log 
+tests/log-tool.sh -s -f $LOG_DIR/simulation.log 
-./log-tool.sh -s -f $LOG_DIR/simulation.log --lastRequest "Browser logout"
+tests/log-tool.sh -s -f $LOG_DIR/simulation.log --lastRequest "Browser logout"
 ```
 
 Set start and end times for the extraction, and create new directory for results:
@@ -34,9 +34,9 @@ mkdir $RESULT_DIR
 
 Extract a portion of the original log, and inspect statistics of resulting log:
 ```
-./log-tool.sh -f $LOG_DIR/simulation.log -o $RESULT_DIR/simulation-$FROM\_$TO.log -e --start $FROM --end $TO 
+tests/log-tool.sh -f $LOG_DIR/simulation.log -o $RESULT_DIR/simulation-$FROM\_$TO.log -e --start $FROM --end $TO 
 
-./log-tool.sh -f $RESULT_DIR/simulation-$FROM\_$TO.log -s
+tests/log-tool.sh -f $RESULT_DIR/simulation-$FROM\_$TO.log -s
 ```
 
 Generate another set of reports from extracted log: 

testsuite/performance/README.md

@@ -1,11 +1,13 @@
 # Keycloak Performance Testsuite
 
 ## Requirements:
+- Bash 2.05+
 - Maven 3.1.1+
 - Keycloak server distribution installed in the local Maven repository. To do this run `mvn install -Pdistribution` from the root of the Keycloak project.
+
+### Docker Compose Provisioner
 - Docker 1.13+
 - Docker Compose 1.14+
-- Bash
 
 ## Getting started for the impatient
 
@@ -47,30 +49,50 @@ Keep reading for more information.
 
 ## Provisioning
 
+### Available provisioners:
+
+- `docker-compose` **Default.** See [`README.docker-compose.md`](README.docker-compose.md) for more details.
+
 ### Provision
 
-Usage: `mvn verify -Pprovision[,cluster] [-D<PARAM>=<VALUE> ...]`. 
+Usage: `mvn verify -Pprovision [-Dprovisioner=<PROVISIONER>] [-D<PARAMETER>=<VALUE>] …`. 
 
-- Single node deployment: `mvn verify -Pprovision`
+#### Deployment Types
-- Cluster deployment: `mvn verify -Pprovision,cluster [-Dkeycloak.scale=N]`. Default `N=1`.
 
-Available parameters are described in [README.provisioning-parameters](README.provisioning-parameters.md).
+- Single node: `mvn verify -Pprovision`
+- Cluster: `mvn verify -Pprovision,cluster [-Dkeycloak.scale=N] [-Dkeycloak.cpusets="cpuset1 cpuset2 … cpusetM"]`. `N ∈ {1 .. M}`.
+- Cross-DC: `mvn verify -Pprovision,crossdc [-Dkeycloak.dc1.scale=K] [-Dkeycloak.dc2.scale=L] [-Dkeycloak.dc1.cpusets=…] [-Dkeycloak.dc2.cpusets=…]`
+
+All available parameters are described in [`README.provisioning-parameters.md`](README.provisioning-parameters.md).
+
+#### Provisioned System
+
+The `provision` operation will produce a `provisioned-system.properties` inside the `tests/target` directory 
+with information about the provisioned system such as the type of deployment and URLs of Keycloak servers and load balancers.
+This information is then used by operations `generate-data`, `import-dump`, `test`, `teardown`.
+
+Provisioning can be run multiple times with different parameters. The system will be updated/reprovisioned based on the new parameters.
+However when switching between different deployment types (e.g. from `singlenode` to `cluster`) it is always necessary 
+to tear down the currently running system.
+
+**Note:** When switching deployment type from `singlenode` or `cluster` to `crossdc` (or the other way around) 
+it is necessary to update the generated Keycloak server configuration (inside `keycloak/target` directory) by 
+adding a `clean` goal to the provisioning command like so: `mvn clean verify -Pprovision …`. It is *not* necessary to update this configuration 
+when switching between `singlenode` and `cluster` deployments.
 
 ### Teardown
 
-Usage: `mvn verify -Pteardown[,cluster]`
+Usage: `mvn verify -Pteardown [-Dprovisioner=<PROVISIONER>]`
 
-- Single node deployment: `mvn verify -Pteardown`
+**Note:** Unless the provisioned system has been properly torn down the maven build will not allow a cleanup of the `tests/target` directory
-- Cluster deployment: `mvn verify -Pteardown,cluster`
+because it contains the `provisioned-system.properties` with information about the still-running system.
-
-Provisioning/teardown is performed via `docker-compose` tool. More details in [README.docker-compose](README.docker-compose.md).
 
 
 ## Testing
 
 ### Generate Test Data
 
-Usage: `mvn verify -Pgenerate-data[,cluster] [-Ddataset=DATASET] [-Dexport-dump] [-D<dataset.property>=<value>]`.
+Usage: `mvn verify -Pgenerate-data [-Ddataset=DATASET] [-D<dataset.property>=<value>]`.
 
 Dataset properties are loaded from `datasets/${dataset}.properties` file. Individual properties can be overriden by specifying `-D` params.
 
@@ -97,7 +119,7 @@ Usage: `mvn verify -Pimport-dump [-Ddataset=DATASET]`
 
 ### Run Tests
 
-Usage: `mvn verify -Ptest[,cluster] [-DrunUsers=N] [-DrampUpPeriod=SECONDS] [-DnumOfIterations=N] [-Ddataset=DATASET] [-D<dataset.property>=<value>]* [-D<test.property>=<value>]* `.
+Usage: `mvn verify -Ptest [-DrunUsers=N] [-DrampUpPeriod=SECONDS] [-DnumOfIterations=N] [-Ddataset=DATASET] [-D<dataset.property>=<value>]* [-D<test.property>=<value>]* `.
 
 _*Note:* The same dataset properties which were used for data generation/import should be supplied to the `test` phase._
 

testsuite/performance/README.provisioning-parameters.md

@@ -1,48 +1,122 @@
 # Keycloak Performance Testsuite - Provisioning Parameters
 
-## Keycloak Server Settings:
+## Overview of Provisioned Services
 
-| Category    | Setting                       | Property                           | Default value                                                    |
+### Testing
-|-------------|-------------------------------|------------------------------------|------------------------------------------------------------------|
-| JVM         | Memory settings               | `keycloak.jvm.memory`              | -Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m   |
-| Undertow    | HTTP Listener max connections | `keycloak.http.max-connections`    | 500                                                              |
-|             | AJP Listener max connections  | `keycloak.ajp.max-connections`     | 500                                                              |
-| IO          | Worker IO thread pool         | `keycloak.worker.io-threads`       | 2                                                                |
-|             | Worker Task thread pool       | `keycloak.worker.task-max-threads` | 16                                                               |
-| Datasources | Connection pool min size      | `keycloak.ds.min-pool-size`        | 10                                                               |
-|             | Connection pool max size      | `keycloak.ds.max-pool-size`        | 100                                                              |
-|             | Connection pool prefill       | `keycloak.ds.pool-prefill`         | true                                                             |
-|             | Prepared statement cache size | `keycloak.ds.ps-cache-size`        | 100                                                              |
 
-## Load Balancer Settings:
+| Deployment      | Keycloak Server                          | Database           | Load Balancer      | Infinispan Server  |
+|-----------------|------------------------------------------|--------------------|--------------------|--------------------|
+| *Singlenode*    | 1 instance                               | 1 instance         | -                  | -                  |
+| *Cluster*       | N instances                              | 1 instance         | 1 instance         | -                  |
+| *Cross-DC*      | K instances in DC1 + L instances in DC2  | 1 instance per DC  | 1 instance per DC  | 1 instance per DC  |
 
-| Category    | Setting                       | Property                              | Default value                                                    |
+### Monitoring
-|-------------|-------------------------------|---------------------------------------|------------------------------------------------------------------|
-| JVM         | Memory settings               | `keycloak-lb.jvm.memory`              | -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m |
-| Undertow    | HTTP Listener max connections | `keycloak-lb.http.max-connections`    | 500                                                              |
-| IO          | Worker IO thread pool         | `keycloak-lb.worker.io-threads`       | 2                                                                |
-|             | Worker Task thread pool       | `keycloak-lb.worker.task-max-threads` | 16                                                               |
 
-## Infinispan Server Settings
+| Deployment      | CAdvisor    | Influx DB   | Grafana     |
+|-----------------|-------------|-------------|-------------|
+| *Monitoring*    | 1 instance  | 1 instance  | 1 instance  |
 
-| Category    | Setting                       | Property                | Default value                                                                           |
-|-------------|-------------------------------|-------------------------|-----------------------------------------------------------------------------------------|
-| JVM         | Memory settings               | `infinispan.jvm.memory` | -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC |
 
-## Docker settings
+## Service Parameters
+
+### Keycloak Server
+
+| Category    | Setting                       | Property                           | Default Value                                                      |
+|-------------|-------------------------------|------------------------------------|--------------------------------------------------------------------|
+| Scaling<sup>[1]</sup> | Scale for cluster   | `keycloak.scale`                   | Maximum size<sup>[2]</sup> of cluster.                             |
+|             | Scale for DC1                 | `keycloak.dc1.scale`               | Maximum size of DC1.                                               |
+|             | Scale for DC2                 | `keycloak.dc2.scale`               | Maximum size of DC2.                                               |
+| Docker      | Allocated CPUs                | `keycloak.docker.cpusets`          | `2-3` for singlenode, `2 3` for cluster deployment                 |
+|             | Allocated CPUs for DC1        | `keycloak.dc1.docker.cpusets`      | `2`                                                                |
+|             | Allocated CPUs for DC2        | `keycloak.dc2.docker.cpusets`      | `3`                                                                |
+|             | Available memory              | `keycloak.docker.memlimit`         | `2500m`                                                            |
+| JVM         | Memory settings               | `keycloak.jvm.memory`              | `-Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m`   |
+| Undertow    | HTTP Listener max connections | `keycloak.http.max-connections`    | `50000`                                                            |
+|             | AJP Listener max connections  | `keycloak.ajp.max-connections`     | `50000`                                                            |
+| IO          | Worker IO thread pool         | `keycloak.worker.io-threads`       | `2`                                                                |
+|             | Worker Task thread pool       | `keycloak.worker.task-max-threads` | `16`                                                               |
+| Datasources | Connection pool min size      | `keycloak.ds.min-pool-size`        | `10`                                                               |
+|             | Connection pool max size      | `keycloak.ds.max-pool-size`        | `100`                                                              |
+|             | Connection pool prefill       | `keycloak.ds.pool-prefill`         | `true`                                                             |
+|             | Prepared statement cache size | `keycloak.ds.ps-cache-size`        | `100`                                                              |
+
+**[ 1 ]** The scaling parameters are optional. They can be set within interval from 1 to the maximum cluster size].
+If not set they are automatically set to the maximum size of the cluster (DC1/DC2 respectively).
+
+**[ 2 ]** Maximum cluster size is determined by provisioner-specific parameter such as `keycloak.docker.cpusets` for the default *docker-compose* provisioner.
+The maximum cluster size corresponds to the number of cpusets.
+
+### Database
+
+| Category    | Setting                       | Property                           | Default Value                                                      |
+|-------------|-------------------------------|------------------------------------|--------------------------------------------------------------------|
+| Docker      | Allocated CPUs                | `db.docker.cpusets`                | `1`                                                                |
+|             | Allocated CPUs for DC1        | `db.dc1.docker.cpusets`            | `1`                                                                |
+|             | Allocated CPUs for DC2        | `db.dc2.docker.cpusets`            | `1`                                                                |
+|             | Available memory              | `db.docker.memlimit`               | `2g`                                                               |
+
+### Load Balancer
+
+| Category    | Setting                       | Property                     | Default Value                                                      |
+|-------------|-------------------------------|------------------------------|--------------------------------------------------------------------|
+| Docker      | Allocated CPUs                | `lb.docker.cpusets`          | `1`                                                                |
+|             | Allocated CPUs for DC1        | `lb.dc1.docker.cpusets`      | `1`                                                                |
+|             | Allocated CPUs for DC2        | `lb.dc2.docker.cpusets`      | `1`                                                                |
+|             | Available memory              | `lb.docker.memlimit`         | `1g`                                                               |
+| JVM         | Memory settings               | `lb.jvm.memory`              | `-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m` |
+| Undertow    | HTTP Listener max connections | `lb.http.max-connections`    | `50000`                                                            |
+| IO          | Worker IO thread pool         | `lb.worker.io-threads`       | `2`                                                                |
+|             | Worker Task thread pool       | `lb.worker.task-max-threads` | `16`                                                               |
+
+### Infinispan Server
+
+| Category    | Setting                       | Property                        | Default Value                                                                             |
+|-------------|-------------------------------|---------------------------------|-------------------------------------------------------------------------------------------|
+| Docker      | Allocated CPUs for DC1        | `infinispan.dc1.docker.cpusets` | `1`                                                                                       |
+|             | Allocated CPUs for DC2        | `infinispan.dc2.docker.cpusets` | `1`                                                                                       |
+|             | Available memory              | `infinispan.docker.memlimit`    | `1500m`                                                                                   |
+| JVM         | Memory settings               | `infinispan.jvm.memory`         | `-Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC`   |
+
+### Monitoring
+
+| Category    | Setting                       | Property                    | Default Value   |
+|-------------|-------------------------------|-----------------------------|-----------------|
+| Docker      | Allocated CPUs                | `monitoring.docker.cpusets` | `0`             |
+
+
+## Note on Docker settings
 
 By default, there are 4 CPU cores allocated: core 0 for monitoring, core 1 for database (MariaDB), and cores 2 and 3 for Keycloak server.
 Default memory limits for database and Keycloak server are 2g. The `cpuset` and `memlimit` parameters set here are set to `cpuset` and
 `mem_limit` parameters of docker-compose configuration. See docker-compose documentation for meaning of the values. How to set the parameters
 correctly depends on number of factors - number of cpu cores, NUMA, available memory etc., hence it is out of scope of this document.
 
-| Container   | Setting                       | Property                        | Default value                                         |
+### Example CPU Settings
-|-------------|-------------------------------|---------------------------------|-------------------------------------------------------|
-| Keycloak    | Allocated CPUs                | `keycloak.docker.cpuset`        | 2-3                                                   |
-|             | Allocated CPUs for DC1        | `keycloak.dc1.docker.cpuset`    | 2-3                                                   |
-|             | Allocated CPUs for DC2        | `keycloak.dc2.docker.cpuset`    | 2-3                                                   |
-|             | Available memory              | `keycloak.docker.memlimit`      | 2g                                                    |
-| MariaDB     | Allocated CPUs                | `db.docker.cpuset`              | 1                                                     |
-|             | Available memory              | `db.docker.memlimit`            | 2g                                                    |
-| Monitoring  | Allocated CPUs                | `monitoring.docker.cpuset`      | 0                                                     |
 
+| HW          | Development Machine  | "Fat Box"    |
+|-------------|----------------------|--------------|
+| Cores       | 4                    | 48           |
+| NUMA Nodes  | 0-3                  | 0-23, 24-47  |
+
+#### Cluster
+
+| Setting                            | Development Machine  | "Fat Box"                   |
+|------------------------------------|----------------------|-----------------------------|
+| `monitoring.docker.cpusets`        | 0                    | 0                           |
+| `db.docker.cpusets`                | 1                    | 1                           |
+| `lb.docker.cpusets`                | 1                    | 2                           |
+| `keycloak.docker.cpusets`          | 2-3                  | 3-6 7-10 11-16 … 43-46      |
+
+#### Cross-DC
+
+| Setting                            | Development Machine  | "Fat Box"                      |
+|------------------------------------|----------------------|--------------------------------|
+| `monitoring.docker.cpusets`        | 0                    | 0                              |
+| `db.dc1.docker.cpusets`            | 1                    | 1                              |
+| `lb.dc1.docker.cpusets`            | 1                    | 2                              |
+| `infinispan.dc1.docker.cpusets`    | 1                    | 3                              |
+| `keycloak.dc1.docker.cpusets`      | 2                    | 4-7 8-11 12-15 16-19 20-23     |
+| `db.dc2.docker.cpusets`            | 1                    | 24                             |
+| `lb.dc2.docker.cpusets`            | 1                    | 25                             |
+| `infinispan.dc2.docker.cpusets`    | 1                    | 26                             |
+| `keycloak.dc2.docker.cpusets`      | 3                    | 27-30 31-34 35-38 39-42 43-46  |

testsuite/performance/docker-compose-cluster.yml

@@ -1,100 +0,0 @@
-version: "2.2"
-
-networks:
-    keycloak:
-        ipam:
-            config:
-            - subnet: 10.0.1.0/24
-    loadbalancing:
-        ipam:
-            config:
-            - subnet: 10.0.2.0/24
-
-services:
-    
-    mariadb:
-        build: db/mariadb
-        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
-        cpuset: ${DB_CPUSET:-1}
-        mem_limit: ${DB_MEMLIMIT:-1g}
-        networks:
-            - keycloak
-        environment:
-            MYSQL_ROOT_PASSWORD: root
-            MYSQL_DATABASE: keycloak
-            MYSQL_USER: keycloak
-            MYSQL_PASSWORD: keycloak
-        ports:
-            - "3306:3306"
-            
-    keycloak:
-        build: keycloak
-        image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
-        depends_on:
-            mariadb:
-                condition: service_healthy
-        cpuset: ${KEYCLOAK_CPUSET:-2-3}
-        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
-        networks:
-            - keycloak
-        environment:
-            CONFIGURATION: standalone-ha.xml
-            PUBLIC_SUBNET: 10.0.1.0/24
-            PRIVATE_SUBNET: 10.0.1.0/24
-            MARIADB_HOSTS: mariadb:3306
-            MARIADB_DATABASE: keycloak
-            MARIADB_USER: keycloak
-            MARIADB_PASSWORD: keycloak
-            KEYCLOAK_USER: admin
-            KEYCLOAK_PASSWORD: admin
-
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
-            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
-            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
-            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
-            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
-            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
-        ports:
-            - "8080"
-            - "9990"
-
-    keycloak_lb:
-        build: load-balancer/wildfly-modcluster
-        image: keycloak_test_wildfly_modcluster:${KEYCLOAK_VERSION:-latest}
-        depends_on:
-            keycloak:
-                condition: service_healthy
-        cpus: 1
-        networks:
-            - keycloak
-#            - loadbalancing
-        environment:
-            PRIVATE_SUBNET: 10.0.1.0/24
-#            PUBLIC_SUBNET: 10.0.2.0/24
-            JAVA_OPTS: ${KEYCLOAK_LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_LB_HTTP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_LB_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_LB_WORKER_TASK_MAX_THREADS:-16}
-        ports:
-            - "8080:8080"
-        
-#    advertize_server:
-#        build: load-balancer/debug-advertise
-#        image: keycloak_test_modcluster_advertize_server:${KEYCLOAK_VERSION:-latest}
-#        networks:
-#            - keycloak
-#            - loadbalancing
-#        command: 0.0.0.0 224.0.1.105 23365
-#
-#    advertize_client:
-#        build: 
-#            context: load-balancer/debug-advertise
-#            dockerfile: Dockerfile-client
-#        image: keycloak_test_modcluster_advertize_client:${KEYCLOAK_VERSION:-latest}
-#        networks:
-#            - keycloak
-#            - loadbalancing
-#        command: 224.0.1.105 23364

testsuite/performance/docker-compose-crossdc.yml

@@ -1,236 +0,0 @@
-version: "2.2"
-
-networks:
-    # DC 1
-    dc1_keycloak:
-        ipam:
-            config:
-            - subnet: 10.1.1.0/24
-    # DC 2
-    dc2_keycloak:
-        ipam:
-            config:
-            - subnet: 10.2.1.0/24
-    # cross-DC
-    loadbalancing:
-        ipam:
-            config:
-            - subnet: 10.0.2.0/24
-    # cross-DC
-    db_replication:
-        ipam:
-            config:
-            - subnet: 10.0.3.0/24
-    # cross-DC
-    ispn_replication:
-        ipam:
-            config:
-            - subnet: 10.0.4.0/24
-        
-services:
-
-    infinispan_dc1:
-        build: 
-            context: infinispan
-            args:
-                LOCAL_SITE: dc1
-                REMOTE_SITE: dc2
-        image: keycloak_test_infinispan_dc1:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
-        networks:
-            - ispn_replication
-            - dc1_keycloak
-        environment:
-            PUBLIC_SUBNET: 10.1.1.0/24
-            PRIVATE_SUBNET: 10.0.4.0/24
-            MGMT_USER: admin
-            MGMT_USER_PASSWORD: admin
-            TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600]
-            JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-        ports:
-            - "9991:9990"
-
-    infinispan_dc2:
-        build: 
-            context: infinispan
-            args:
-                LOCAL_SITE: dc2
-                REMOTE_SITE: dc1
-        image: keycloak_test_infinispan_dc2:${KEYCLOAK_VERSION:-latest}
-        depends_on:
-            infinispan_dc1:
-                condition: service_healthy
-        cpus: 1
-        networks:
-            - ispn_replication
-            - dc2_keycloak
-        environment:
-            PUBLIC_SUBNET: 10.2.1.0/24
-            PRIVATE_SUBNET: 10.0.4.0/24
-            MGMT_USER: admin
-            MGMT_USER_PASSWORD: admin
-            TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600],infinispan_dc2[7600]
-            JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-        ports:
-            - "9992:9990"
-    
-    
-    mariadb_dc1:
-        build: db/mariadb
-        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
-        networks:
-            - db_replication
-            - dc1_keycloak
-        environment:
-            MYSQL_ROOT_PASSWORD: root
-            MYSQL_INITDB_SKIP_TZINFO: foo
-            MYSQL_DATABASE: keycloak
-            MYSQL_USER: keycloak
-            MYSQL_PASSWORD: keycloak
-        entrypoint: docker-entrypoint-wsrep.sh
-        command: --wsrep-new-cluster
-        ports:
-            - "3306:3306"
-            
-    mariadb_dc2:
-        build: db/mariadb
-        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
-        depends_on: 
-            mariadb_dc1:
-                condition: service_healthy
-        cpuset: ${DB_CPUSET:-1}
-        mem_limit: ${DB_MEMLIMIT:-1g}
-        networks:
-            - db_replication
-            - dc2_keycloak
-        environment:
-            MYSQL_ROOT_PASSWORD: root
-            MYSQL_INITDB_SKIP_TZINFO: foo
-        entrypoint: docker-entrypoint-wsrep.sh
-        command: --wsrep_cluster_address=gcomm://mariadb_dc1
-        ports:
-            - "3307:3306"
-
-
-
-    keycloak_dc1:
-        build: 
-            context: ./keycloak
-            args:
-                REMOTE_CACHES: "true"
-        image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
-        depends_on:
-            # wait for the db cluster to be ready before starting keycloak
-            mariadb_dc2:
-                condition: service_healthy
-            # wait for the ispn cluster to be ready before starting keycloak
-            infinispan_dc2:
-                condition: service_healthy
-        cpuset: ${KEYCLOAK_DC1_CPUSET:-2}
-        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
-        networks:
-            - dc1_keycloak
-        environment:
-            CONFIGURATION: standalone-ha.xml
-            PUBLIC_SUBNET: 10.1.1.0/24
-            PRIVATE_SUBNET: 10.1.1.0/24
-            MARIADB_HOSTS: mariadb_dc1:3306
-            MARIADB_DATABASE: keycloak
-            MARIADB_USER: keycloak
-            MARIADB_PASSWORD: keycloak
-            KEYCLOAK_USER: admin
-            KEYCLOAK_PASSWORD: admin
-            INFINISPAN_HOST: infinispan_dc1
-            SITE: dc1
-            
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
-            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
-            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
-            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
-            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
-            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
-        ports:
-            - "8080"
-            - "9990"
-
-
-    keycloak_dc2:
-        build: 
-            context: ./keycloak
-            args:
-                REMOTE_CACHES: "true"
-        image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
-        depends_on:
-            # wait for first kc instance to be ready before starting another
-            keycloak_dc1:
-                condition: service_healthy
-        cpuset: ${KEYCLOAK_DC2_CPUSET:-3}
-        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
-        networks:
-            - dc2_keycloak
-        environment:
-            CONFIGURATION: standalone-ha.xml
-            PUBLIC_SUBNET: 10.2.1.0/24
-            PRIVATE_SUBNET: 10.2.1.0/24
-            MARIADB_HOSTS: mariadb_dc2:3306
-            MARIADB_DATABASE: keycloak
-            MARIADB_USER: keycloak
-            MARIADB_PASSWORD: keycloak
-            INFINISPAN_HOST: infinispan_dc2
-            SITE: dc2
-
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
-            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
-            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
-            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
-            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
-            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
-        ports:
-            - "8080"
-            - "9990"
-
-
-
-
-
-    keycloak_lb_dc1:
-        build: load-balancer/wildfly-modcluster
-        image: keycloak_test_keycloak_lb:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
-        networks:
-            - dc1_keycloak
-#            - loadbalancing
-        environment:
-            PRIVATE_SUBNET: 10.1.1.0/24
-#            PUBLIC_SUBNET: 10.0.2.0/24
-            JAVA_OPTS: ${KEYCLOAK_LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_LB_HTTP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_LB_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_LB_WORKER_TASK_MAX_THREADS:-16}
-        ports:
-            - "8081:8080"
-            
-    keycloak_lb_dc2:
-        build: load-balancer/wildfly-modcluster
-        image: keycloak_test_keycloak_lb:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
-        networks:
-            - dc2_keycloak
-#            - loadbalancing
-        environment:
-            PRIVATE_SUBNET: 10.2.1.0/24
-#            PUBLIC_SUBNET: 10.0.2.0/24
-            JAVA_OPTS: ${KEYCLOAK_LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_LB_HTTP_MAX_CONNECTIONS:-500}
-            WORKER_IO_THREADS: ${KEYCLOAK_LB_WORKER_IO_THREADS:-2}
-            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_LB_WORKER_TASK_MAX_THREADS:-16}
-        ports:
-            - "8082:8080"
-            

testsuite/performance/docker-compose-db-failover.yml

@@ -71,7 +71,7 @@ services:
 
 
     keycloak:
-        build: keycloak
+        build: keycloak/target/docker
         image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
         networks:
             - keycloak

testsuite/performance/healthcheck.sh

@@ -1,34 +0,0 @@
-#!/bin/bash
-
-function isKeycloakHealthy {
-    echo Checking instance $1
-    CODE=`curl -s -o /dev/null -w "%{http_code}" $1/realms/master`
-    [ "$CODE" -eq "200" ]
-}
-
-function waitUntilSystemHealthy {
-    echo Waiting until all Keycloak instances are healthy...
-    if [ -z $1 ]; then ITERATIONS=60; else ITERATIONS=$1; fi
-    C=0
-    SYSTEM_HEALTHY=false
-    while ! $SYSTEM_HEALTHY; do 
-        C=$((C+1))
-        if [ $C -gt $ITERATIONS ]; then
-            echo System healthcheck failed.
-            exit 1
-        fi
-        sleep 2
-        SYSTEM_HEALTHY=true
-        for URI in $KEYCLOAK_SERVER_URIS ; do
-            if ! isKeycloakHealthy $URI; then 
-                echo Instance is not healthy.
-                SYSTEM_HEALTHY=false
-            fi
-        done
-    done
-    echo System is healthy.
-}
-
-if [ -z "$KEYCLOAK_SERVER_URIS" ]; then KEYCLOAK_SERVER_URIS=http://localhost:8080/auth; fi
-
-waitUntilSystemHealthy

testsuite/performance/keycloak/Dockerfile

@@ -1,40 +0,0 @@
-FROM jboss/base-jdk:8
-
-ENV JBOSS_HOME /opt/jboss/keycloak
-ARG REMOTE_CACHES=false
-WORKDIR $JBOSS_HOME
-
-ENV CONFIGURATION standalone.xml
-ENV DEBUG_USER admin
-ENV DEBUG_USER_PASSWORD admin
-
-# Enables signals getting passed from startup script to JVM
-# ensuring clean shutdown when container is stopped.
-ENV LAUNCH_JBOSS_IN_BACKGROUND 1
-ENV PROXY_ADDRESS_FORWARDING false
-
-ADD target/keycloak configs/ ./
-ADD *.sh /usr/local/bin/
-
-USER root
-RUN chown -R jboss .; chgrp -R jboss .; \
-    chmod -R -v +x /usr/local/bin/ ; \
-    yum install -y epel-release jq iproute && yum clean all
-
-USER jboss
-# install mariadb JDBC driver
-RUN curl --create-dirs --output modules/system/layers/base/org/mariadb/jdbc/main/mariadb-java-client-2.0.3.jar http://central.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.0.3/mariadb-java-client-2.0.3.jar ; \
-    $JBOSS_HOME/bin/jboss-cli.sh --file=set-keycloak-ds.cli && \
-    $JBOSS_HOME/bin/jboss-cli.sh --file=io-worker-threads.cli && \
-    $JBOSS_HOME/bin/jboss-cli.sh --file=undertow.cli && \
-    $JBOSS_HOME/bin/jboss-cli.sh --file=modcluster-simple-load-provider.cli && \
-    if [ "$REMOTE_CACHES" == "true" ]; then $JBOSS_HOME/bin/jboss-cli.sh --file=add-remote-cache-stores.cli; fi && \
-    cd $JBOSS_HOME/standalone; rm -rf configuration/standalone_xml_history log data tmp ; \
-    $JBOSS_HOME/bin/add-user.sh -u $DEBUG_USER -p $DEBUG_USER_PASSWORD
-
-ADD module.xml modules/system/layers/base/org/mariadb/jdbc/main/
-
-EXPOSE 8080
-EXPOSE 9990
-HEALTHCHECK  --interval=5s --timeout=5s --retries=12 CMD ["keycloak-healthcheck.sh"]
-ENTRYPOINT ["docker-entrypoint.sh"]

testsuite/performance/keycloak/configure.xml

@@ -0,0 +1,79 @@
+<project name="keycloak-server-configuration" basedir="." >
+
+    <target name="check-if-performance-configured">
+        <available property="performance.configured" file="${project.build.directory}/performance-configured"/>
+        <echo>performance.configured: ${performance.configured}</echo>
+    </target>                            
+    
+    <target name="keycloak-performance-configuration" unless="performance.configured" depends="check-if-performance-configured">
+        <echo>keycloak-performance-configuration</echo>
+        <chmod perm="ug+x">
+            <fileset dir="${server.unpacked.home}/bin">
+                <include name="*.sh"/>
+            </fileset>
+        </chmod>
+        <filter token="MODULE_NAME" value="${jdbc.driver.groupId}"/>
+        <filter token="RESOURCE_ROOT_PATH" value="${jdbc.driver.artifactId}-${jdbc.driver.version}.jar"/>
+        <copy file="${resources.dir}/module.xml"
+              todir="${server.unpacked.home}/modules/system/layers/base/${jdbc.driver.module.path}/main"
+              filtering="true"
+        />
+        <copy todir="${server.unpacked.home}/bin" >
+            <fileset dir="${scripts.dir}/jboss-cli"/>
+        </copy>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=set-keycloak-ds.cli"/>
+        </exec>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=io-worker-threads.cli"/>
+        </exec>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=undertow.cli"/>
+        </exec>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=modcluster-simple-load-provider.cli"/>
+        </exec>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=io-worker-threads.cli"/>
+        </exec>
+        <exec executable="./${add.user.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="-u"/>
+            <arg value="${keycloak.debug.user}"/>
+            <arg value="-p"/>
+            <arg value="${keycloak.debug.user.password}"/>
+        </exec>
+        <delete dir="${server.unpacked.home}/standalone/configuration/standalone_xml_history"/>
+        <touch file="${project.build.directory}/performance-configured"/>
+    </target>
+
+        
+    <target name="check-if-crossdc-configured">
+        <available property="crossdc.configured" file="${project.build.directory}/crossdc-configured"/>
+        <echo>crossdc.configured: ${crossdc.configured}</echo>
+    </target>         
+    
+    <target name="keycloak-crossdc-configuration" unless="crossdc.configured" depends="check-if-crossdc-configured">
+        <echo>keycloak-crossdc-configuration</echo>
+        <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
+            <arg value="--file=add-remote-cache-stores.cli"/>
+        </exec>
+        <delete dir="${server.unpacked.home}/standalone/configuration/standalone_xml_history"/>
+        <touch file="${project.build.directory}/crossdc-configured"/>
+    </target>
+
+
+    <target name="keycloak-docker">
+        <copy todir="${project.build.directory}/docker" overwrite="false">
+            <fileset dir="${scripts.dir}">
+                <include name="Dockerfile"/>
+                <include name="*.sh"/>
+            </fileset>
+        </copy>
+        <copy todir="${project.build.directory}/docker/keycloak" overwrite="false">
+            <fileset dir="${server.unpacked.home}">
+                <exclude name="bin/*.cli"/>
+            </fileset>
+        </copy>
+    </target>
+
+</project>

testsuite/performance/keycloak/pom.xml

@@ -37,7 +37,27 @@
     <properties>
         <server.groupId>org.keycloak</server.groupId>
         <server.artifactId>keycloak-server-dist</server.artifactId>
-        <server.unpacked.folder>keycloak-${product.version}</server.unpacked.folder>
+        <server.unpacked.home>${project.build.directory}/keycloak-${product.version}</server.unpacked.home>
+        
+        <jdbc.driver.groupId>org.mariadb.jdbc</jdbc.driver.groupId>
+        <jdbc.driver.artifactId>mariadb-java-client</jdbc.driver.artifactId>
+        <jdbc.driver.version>2.0.3</jdbc.driver.version>
+        <jdbc.driver.module.path>org/mariadb/jdbc</jdbc.driver.module.path>
+
+        <script.extension>sh</script.extension>
+        <jboss.cli.script>jboss-cli.${script.extension}</jboss.cli.script>
+        <add.user.script>add-user.${script.extension}</add.user.script>
+        
+        <skip.crossdc.configuration>true</skip.crossdc.configuration>
+        
+        <skip.configuration>false</skip.configuration>
+        <skip.keycloak.docker>false</skip.keycloak.docker>
+        
+        <keycloak.debug.user>admin</keycloak.debug.user>
+        <keycloak.debug.user.password>admin</keycloak.debug.user.password>
+        
+        <scripts.dir>${project.build.scriptSourceDirectory}</scripts.dir>
+        <resources.dir>${project.basedir}/src/main/resources</resources.dir>
     </properties>
 
     <build>
@@ -47,14 +67,13 @@
                 <artifactId>maven-dependency-plugin</artifactId>
                 <executions>
                     <execution>
-                        <!--=============================-->
                         <id>unpack-keycloak-server-dist</id>
-                        <!--=============================-->
                         <phase>generate-resources</phase>
                         <goals>
                             <goal>unpack</goal>
                         </goals>
                         <configuration>
+                            <overWriteIfNewer>true</overWriteIfNewer>
                             <artifactItems>
                                 <artifactItem>
                                     <groupId>${server.groupId}</groupId>
@@ -66,6 +85,25 @@
                             </artifactItems>
                         </configuration>
                     </execution>
+                    <execution>
+                        <id>copy-jdbc-driver</id>
+                        <phase>generate-resources</phase>
+                        <goals>
+                            <goal>copy</goal>
+                        </goals>
+                        <configuration>
+                            <overWriteIfNewer>true</overWriteIfNewer>
+                            <artifactItems>
+                                <artifactItem>
+                                    <groupId>${jdbc.driver.groupId}</groupId>
+                                    <artifactId>${jdbc.driver.artifactId}</artifactId>
+                                    <version>${jdbc.driver.version}</version>
+                                    <type>jar</type>
+                                    <outputDirectory>${server.unpacked.home}/modules/system/layers/base/${jdbc.driver.module.path}/main</outputDirectory>
+                                </artifactItem>
+                            </artifactItems>
+                        </configuration>
+                    </execution>
                 </executions>
             </plugin>
             <plugin>
@@ -73,18 +111,40 @@
                 <artifactId>maven-antrun-plugin</artifactId>
                 <executions>
                     <execution>
-                        <!--=============================-->
+                        <id>keycloak-performance-configuration</id>
-                        <id>rename-keycloak-folder</id>
-                        <!--=============================-->
                         <phase>process-resources</phase>
                         <goals>
                             <goal>run</goal>
                         </goals>
                         <configuration>
                             <target>
-                                <move todir="${project.build.directory}/keycloak" failonerror="false">
+                                <ant antfile="configure.xml" target="keycloak-performance-configuration" />
-                                    <fileset dir="${project.build.directory}/${server.unpacked.folder}"/>
+                            </target>
-                                </move>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>keycloak-crossdc-configuration</id>
+                        <phase>process-resources</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <skip>${skip.crossdc.configuration}</skip>
+                            <target>
+                                <ant antfile="configure.xml" target="keycloak-crossdc-configuration" />
+                            </target>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>keycloak-docker</id>
+                        <phase>process-resources</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <skip>${skip.keycloak.docker}</skip>
+                            <target>
+                                <ant antfile="configure.xml" target="keycloak-docker" />
                             </target>
                         </configuration>
                     </execution>
@@ -95,14 +155,34 @@
     
     <profiles>
         
+        <profile>        
+            <id>windows</id>
+            <activation>
+                <os>
+                    <family>windows</family>
+                </os>
+            </activation>
+            <properties>
+                <script.extension>ps1</script.extension>
+            </properties>
+        </profile>
+        
+        <profile>
+            <id>crossdc</id>
+            <properties>
+                <skip.crossdc.configuration>false</skip.crossdc.configuration>
+            </properties>
+        </profile>
+        
         <profile>
             <id>integration-testsuite-server</id>
             <properties>
                 <server.groupId>org.keycloak.testsuite</server.groupId>
                 <server.artifactId>integration-arquillian-servers-auth-server-wildfly</server.artifactId>
-                <server.unpacked.folder>auth-server-wildfly</server.unpacked.folder>
+                <server.unpacked.home>auth-server-wildfly</server.unpacked.home>
             </properties>
         </profile>
+
     </profiles>
     
 </project>

testsuite/performance/keycloak/src/main/resources/module.xml

@@ -20,9 +20,9 @@
   ~ Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
   ~ 02110-1301 USA, or see the FSF site: http://www.fsf.org.
   -->
-<module xmlns="urn:jboss:module:1.0" name="org.mariadb.jdbc">
+<module xmlns="urn:jboss:module:1.0" name="@MODULE_NAME@">
    <resources>
-      <resource-root path="mariadb-java-client-2.0.3.jar"/>
+      <resource-root path="@RESOURCE_ROOT_PATH@"/>
    </resources>
    <dependencies>
       <module name="javax.api"/>

testsuite/performance/keycloak/src/main/scripts/Dockerfile

@@ -0,0 +1,28 @@
+FROM jboss/base-jdk:8
+
+ENV JBOSS_HOME /opt/jboss/keycloak
+WORKDIR $JBOSS_HOME
+
+ENV CONFIGURATION standalone.xml
+
+# Enables signals getting passed from startup script to JVM
+# ensuring clean shutdown when container is stopped.
+ENV LAUNCH_JBOSS_IN_BACKGROUND 1
+ENV PROXY_ADDRESS_FORWARDING false
+
+USER root
+RUN yum install -y epel-release jq iproute && yum clean all
+
+ADD keycloak ./
+ADD *.sh /usr/local/bin/
+
+USER root
+RUN chown -R jboss .; chgrp -R jboss .; chmod -R -v ug+x bin/*.sh ; \
+    chmod -R -v ug+x /usr/local/bin/ 
+
+USER jboss
+
+EXPOSE 8080
+EXPOSE 9990
+HEALTHCHECK  --interval=5s --timeout=5s --retries=12 CMD ["keycloak-healthcheck.sh"]
+ENTRYPOINT ["docker-entrypoint.sh"]

testsuite/performance/keycloak/src/main/scripts/jboss-cli/undertow-batch.cli

@@ -1,2 +1,2 @@
 /subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding, value=${env.PROXY_ADDRESS_FORWARDING})
-/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=max-connections,value=${env.HTTP_MAX_CONNECTIONS:500})
+/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=max-connections,value=${env.HTTP_MAX_CONNECTIONS:50000})

testsuite/performance/keycloak/src/main/scripts/jboss-cli/undertow.cli

@@ -6,4 +6,4 @@ stop-embedded-server
 embed-server --server-config=standalone-ha.xml
 run-batch --file=undertow-batch.cli
 
-/subsystem=undertow/server=default-server/ajp-listener=ajp:write-attribute(name=max-connections,value=${env.AJP_MAX_CONNECTIONS:500})
+/subsystem=undertow/server=default-server/ajp-listener=ajp:write-attribute(name=max-connections,value=${env.AJP_MAX_CONNECTIONS:50000})

testsuite/performance/load-dump.sh

@@ -1,26 +0,0 @@
-#!/bin/bash
-DIRNAME=`dirname "$0"`
-GATLING_HOME=$DIRNAME/tests
-
-if [ -z "$DATASET" ]; then
-    echo "Please specify DATASET env variable.";
-    exit 1;
-fi
-
-if [ ! -f $GATLING_HOME/datasets/$DATASET.sql ]; then
-    if [ ! -f $GATLING_HOME/datasets/$DATASET.sql.gz ]; then
-        echo "Data dump file does not exist: $GATLING_HOME/datasets/$DATASET.sql(.gz)";
-        echo "Usage: DATASET=[name] $0"
-        exit 1;
-    else
-        FILE=$GATLING_HOME/datasets/$DATASET.sql.gz
-        CMD="zcat $FILE"
-    fi
-else
-    FILE=$GATLING_HOME/datasets/$DATASET.sql
-    CMD="cat $FILE"
-fi
-
-echo "Importing dump file: $FILE"
-echo "\$ $CMD | docker exec -i performance_mariadb_1 /usr/bin/mysql -u root --password=root keycloak"
-$CMD | docker exec -i performance_mariadb_1 /usr/bin/mysql -u root --password=root keycloak

testsuite/performance/log-tool.sh

@@ -1,5 +0,0 @@
-#!/bin/bash
-DIRNAME=`dirname "$0"`
-GATLING_HOME=$DIRNAME/tests
-
-java -cp $GATLING_HOME/target/classes org.keycloak.performance.log.LogProcessor "$@"

testsuite/performance/monitoring/cadvisor/Dockerfile

@@ -1,4 +1,5 @@
 FROM google/cadvisor:v0.26.1
 RUN apk add --no-cache bash curl
 ADD entrypoint.sh /entrypoint.sh
+RUN chmod +x -v /entrypoint.sh
 ENTRYPOINT ["/entrypoint.sh"]

testsuite/performance/monitoring/grafana/Dockerfile

@@ -4,4 +4,5 @@ ENV GF_DASHBOARDS_JSON_PATH /etc/grafana/dashboards/
 COPY resource-usage-per-container.json /etc/grafana/dashboards/
 COPY resource-usage-combined.json /etc/grafana/dashboards/
 ADD entrypoint.sh /entrypoint.sh
+RUN chmod +x -v /entrypoint.sh
 ENTRYPOINT ["/entrypoint.sh"]

testsuite/performance/prepare-dump.sh

@@ -1,18 +0,0 @@
-#!/bin/bash
-DIRNAME=`dirname "$0"`
-GATLING_HOME=$DIRNAME/tests
-
-if [ -z "$DATASET" ]; then
-    echo "This script requires DATASET env variable to be set"
-    exit 1
-fi
-
-echo "Exporting dump file"
-docker exec performance_mariadb_1 /usr/bin/mysqldump -u root --password=root keycloak > $DATASET.sql
-if [ $? -ne 0 ]; then
-    echo "Failed!"
-    exit $?
-fi
-
-gzip $DATASET.sql
-mv $DATASET.sql.gz $GATLING_HOME/datasets/

testsuite/performance/tests/common.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+PROJECT_BASEDIR=${PROJECT_BASEDIR:-$(cd "$(dirname "$0")"; pwd)}
+PROJECT_BUILD_DIRECTORY=$PROJECT_BASEDIR/target
+
+export PROVISIONED_SYSTEM_PROPERTIES_FILE="$PROJECT_BUILD_DIRECTORY/provisioned-system.properties"

testsuite/performance/tests/datasets/200ku200c.properties

@@ -0,0 +1,8 @@
+numOfRealms=1
+usersPerRealm=200000
+clientsPerRealm=200
+realmRoles=2
+realmRolesPerUser=2
+clientRolesPerUser=2
+clientRolesPerClient=2
+hashIterations=27500

testsuite/performance/tests/datasets/500ku500c.properties

@@ -0,0 +1,8 @@
+numOfRealms=1
+usersPerRealm=500000
+clientsPerRealm=500
+realmRoles=2
+realmRolesPerUser=2
+clientRolesPerUser=2
+clientRolesPerClient=2
+hashIterations=27500

testsuite/performance/tests/docker-compose.sh

@@ -0,0 +1,371 @@
+#!/bin/bash
+
+### FUNCTIONS ###
+
+function runCommand() {
+    echo "$1"
+    if ! eval "$1" ; then 
+        echo "Execution of command failed."
+        echo "Command: \"$1\""
+        exit 1;
+    fi
+}
+
+function generateDockerComposeFile() {
+    echo "Generating $( basename "$DOCKER_COMPOSE_FILE" )"
+    local TEMPLATES_PATH=$DEPLOYMENT
+    cat $TEMPLATES_PATH/docker-compose-base.yml > $DOCKER_COMPOSE_FILE
+    case "$DEPLOYMENT" in
+        cluster)
+            I=0
+            for CPUSET in $KEYCLOAK_CPUSETS ; do
+                I=$((I+1))
+                sed -e s/%I%/$I/ -e s/%CPUSET%/$CPUSET/ $TEMPLATES_PATH/docker-compose-keycloak.yml >> $DOCKER_COMPOSE_FILE
+            done
+        ;;
+        crossdc) 
+            I=0
+            for CPUSET in $KEYCLOAK_DC1_CPUSETS ; do
+                I=$((I+1))
+                sed -e s/%I%/$I/ -e s/%CPUSET%/$CPUSET/ $TEMPLATES_PATH/docker-compose-keycloak_dc1.yml >> $DOCKER_COMPOSE_FILE
+            done
+            I=0
+            for CPUSET in $KEYCLOAK_DC2_CPUSETS ; do
+                I=$((I+1))
+                sed -e s/%I%/$I/ -e s/%CPUSET%/$CPUSET/ $TEMPLATES_PATH/docker-compose-keycloak_dc2.yml >> $DOCKER_COMPOSE_FILE
+            done
+        ;;
+    esac
+}
+
+function inspectDockerPortMapping() {
+    local PORT="$1"
+    local CONTAINER="$2"
+    local INSPECT_COMMAND="docker inspect --format='{{(index (index .NetworkSettings.Ports \"$PORT\") 0).HostPort}}' $CONTAINER"
+    MAPPED_PORT="$( eval "$INSPECT_COMMAND" )"
+    if [ -z "$MAPPED_PORT" ]; then 
+        echo "Error finding mapped port for $CONTAINER."
+        exit 1
+    fi
+}
+
+function generateProvisionedSystemProperties() {
+    echo "Generating $PROVISIONED_SYSTEM_PROPERTIES_FILE"
+    echo "deployment=$DEPLOYMENT" > $PROVISIONED_SYSTEM_PROPERTIES_FILE
+    echo "keycloak.docker.services=$KEYCLOAK_SERVICES" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+    case "$DEPLOYMENT" in
+        singlenode)
+            inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_keycloak_1
+            echo "keycloak.frontend.servers=http://localhost:$MAPPED_PORT/auth" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+        ;;
+        cluster)
+            inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_1
+            echo "keycloak.frontend.servers=http://localhost:$MAPPED_PORT/auth" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            BACKEND_URLS=""
+            for SERVICE in $KEYCLOAK_SERVICES ; do
+                inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_${SERVICE}_1
+                BACKEND_URLS="$BACKEND_URLS http://localhost:$MAPPED_PORT/auth"
+            done
+            echo "keycloak.backend.servers=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+        ;;
+        crossdc) 
+            inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_dc1_1
+            KC_DC1_PORT=$MAPPED_PORT
+            inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_dc2_1
+            KC_DC2_PORT=$MAPPED_PORT
+            echo "keycloak.frontend.servers=http://localhost:$KC_DC1_PORT/auth http://localhost:$KC_DC2_PORT/auth" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            BACKEND_URLS=""
+            for SERVICE in $KEYCLOAK_SERVICES ; do
+                inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_${SERVICE}_1
+                BACKEND_URLS="$BACKEND_URLS http://localhost:$MAPPED_PORT/auth"
+            done
+            echo "keycloak.backend.servers=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+        ;;
+    esac
+}
+
+function loadProvisionedSystemProperties() {
+    if [ -f $PROVISIONED_SYSTEM_PROPERTIES_FILE ]; then 
+        echo "Loading $PROVISIONED_SYSTEM_PROPERTIES_FILE"
+        export DEPLOYMENT=$( grep -Po "(?<=^deployment=).*" $PROVISIONED_SYSTEM_PROPERTIES_FILE )
+        export KEYCLOAK_SERVICES=$( grep -Po "(?<=^keycloak.docker.services=).*" $PROVISIONED_SYSTEM_PROPERTIES_FILE )
+    else
+        echo "$PROVISIONED_SYSTEM_PROPERTIES_FILE not found."
+    fi
+}
+
+function removeProvisionedSystemProperties() {
+    rm -f $PROVISIONED_SYSTEM_PROPERTIES_FILE
+}
+
+function isTestDeployment() {
+    IS_TEST_DEPLOYMENT=false
+    case "$DEPLOYMENT" in
+        singlenode|cluster|crossdc) IS_TEST_DEPLOYMENT=true ;;
+    esac
+    $IS_TEST_DEPLOYMENT
+}
+
+function validateNotEmpty() {
+    VARIABLE_NAME=$1
+    VARIABLE_VALUE=$2
+#    echo "$VARIABLE_NAME: $VARIABLE_VALUE"
+    if [ -z "$VARIABLE_VALUE" ]; then echo "$VARIABLE_NAME must contain at least one item."; exit 1; fi
+}
+
+
+### SCRIPT ###
+
+cd "$(dirname "$0")"
+. ./common.sh
+
+cd $PROJECT_BUILD_DIRECTORY/docker-compose
+
+PROJECT_NAME=performance
+
+
+export DEPLOYMENT="${DEPLOYMENT:-singlenode}"
+if isTestDeployment ; then loadProvisionedSystemProperties; fi
+export OPERATION="${OPERATION:-provision}"
+
+echo "DEPLOYMENT: $DEPLOYMENT"
+
+case "$DEPLOYMENT" in
+    singlenode) DOCKER_COMPOSE_FILE=docker-compose.yml ;;
+    cluster) DOCKER_COMPOSE_FILE=docker-compose-cluster.yml ;;
+    crossdc) DOCKER_COMPOSE_FILE=docker-compose-crossdc.yml ;;
+    monitoring) DOCKER_COMPOSE_FILE=docker-compose-monitoring.yml ; DELETE_DATA="${DELETE_DATA:-false}" ;;
+    *)
+        echo "Deployment '$DEPLOYMENT' not supported by provisioner '$PROVISIONER'."
+        exit 1
+    ;;
+esac
+
+
+echo "OPERATION: $OPERATION"
+
+case "$OPERATION" in
+
+    provision)
+
+        case "$DEPLOYMENT" in
+
+            singlenode)
+                BASE_SERVICES="mariadb"
+                KEYCLOAK_SERVICES="keycloak"
+
+                validateNotEmpty DB_CPUSETS $DB_CPUSETS
+                DB_CPUSETS_ARRAY=( $DB_CPUSETS )
+                DB_CPUSET=${DB_CPUSETS_ARRAY[0]}
+
+                validateNotEmpty KEYCLOAK_CPUSETS $KEYCLOAK_CPUSETS
+                KEYCLOAK_CPUSETS_ARRAY=( $KEYCLOAK_CPUSETS )
+                KEYCLOAK_CPUSET=${KEYCLOAK_CPUSETS_ARRAY[0]}
+
+                echo "DB_CPUSET: $DB_CPUSET"
+                echo "KEYCLOAK_CPUSET: $KEYCLOAK_CPUSET"
+                echo "BASE_SERVICES: $BASE_SERVICES"
+                echo "KEYCLOAK_SERVICES: $KEYCLOAK_SERVICES"
+
+            ;;
+
+            cluster)
+                BASE_SERVICES="mariadb loadbalancer"
+
+                validateNotEmpty DB_CPUSETS $DB_CPUSETS
+                DB_CPUSETS_ARRAY=( $DB_CPUSETS )
+                DB_CPUSET=${DB_CPUSETS_ARRAY[0]}
+
+                validateNotEmpty LB_CPUSETS $LB_CPUSETS
+                LB_CPUSETS_ARRAY=( $LB_CPUSETS )
+                LB_CPUSET=${LB_CPUSETS_ARRAY[0]}
+
+                validateNotEmpty KEYCLOAK_CPUSETS $KEYCLOAK_CPUSETS
+                KEYCLOAK_CPUSETS_ARRAY=( $KEYCLOAK_CPUSETS )
+
+                KEYCLOAK_MAX_SCALE=${#KEYCLOAK_CPUSETS_ARRAY[@]}
+                KEYCLOAK_SCALE="${KEYCLOAK_SCALE:-$KEYCLOAK_MAX_SCALE}"
+                if [ $KEYCLOAK_SCALE -gt $KEYCLOAK_MAX_SCALE ]; then KEYCLOAK_SCALE=$KEYCLOAK_MAX_SCALE; fi
+                if [ $KEYCLOAK_SCALE -lt 1 ]; then KEYCLOAK_SCALE=1; fi
+
+                echo "DB_CPUSET: $DB_CPUSET"
+                echo "KEYCLOAK_CPUSETS: $KEYCLOAK_CPUSETS"
+                echo "KEYCLOAK_SCALE: ${KEYCLOAK_SCALE} (max ${KEYCLOAK_MAX_SCALE})"
+
+                KEYCLOAK_SERVICES=""
+                STOPPED_KEYCLOAK_SERVICES=""
+                for ((i=1; i<=$KEYCLOAK_MAX_SCALE; i++)) ; do
+                    if (( $i <= $KEYCLOAK_SCALE )) ; then
+                        KEYCLOAK_SERVICES="$KEYCLOAK_SERVICES keycloak_$i"
+                    else
+                        STOPPED_KEYCLOAK_SERVICES="$STOPPED_KEYCLOAK_SERVICES keycloak_$i"
+                    fi
+                done
+                echo "BASE_SERVICES: $BASE_SERVICES"
+                echo "KEYCLOAK_SERVICES: $KEYCLOAK_SERVICES"
+                
+                generateDockerComposeFile
+            ;;
+
+            crossdc)
+                BASE_SERVICES="mariadb_dc1 mariadb_dc2 infinispan_dc1 infinispan_dc2 loadbalancer_dc1 loadbalancer_dc2"
+
+                validateNotEmpty DB_DC1_CPUSETS $DB_DC1_CPUSETS
+                validateNotEmpty DB_DC2_CPUSETS $DB_DC2_CPUSETS
+                DB_DC1_CPUSETS_ARRAY=( $DB_DC1_CPUSETS )
+                DB_DC2_CPUSETS_ARRAY=( $DB_DC2_CPUSETS )
+                DB_DC1_CPUSET=${DB_DC1_CPUSETS_ARRAY[0]}
+                DB_DC2_CPUSET=${DB_DC2_CPUSETS_ARRAY[0]}
+                echo "DB_DC1_CPUSET: $DB_DC1_CPUSET"
+                echo "DB_DC2_CPUSET: $DB_DC2_CPUSET"
+
+                validateNotEmpty LB_DC1_CPUSETS $LB_DC1_CPUSETS
+                validateNotEmpty LB_DC2_CPUSETS $LB_DC2_CPUSETS
+                LB_DC1_CPUSETS_ARRAY=( $LB_DC1_CPUSETS )
+                LB_DC2_CPUSETS_ARRAY=( $LB_DC2_CPUSETS )
+                LB_DC1_CPUSET=${LB_DC1_CPUSETS_ARRAY[0]}
+                LB_DC2_CPUSET=${LB_DC2_CPUSETS_ARRAY[0]}
+                echo "LB_DC1_CPUSET: $LB_DC1_CPUSET"
+                echo "LB_DC2_CPUSET: $LB_DC2_CPUSET"
+
+                validateNotEmpty INFINISPAN_DC1_CPUSETS $INFINISPAN_DC1_CPUSETS
+                validateNotEmpty INFINISPAN_DC2_CPUSETS $INFINISPAN_DC2_CPUSETS
+                INFINISPAN_DC1_CPUSETS_ARRAY=( $INFINISPAN_DC1_CPUSETS )
+                INFINISPAN_DC2_CPUSETS_ARRAY=( $INFINISPAN_DC2_CPUSETS )
+                INFINISPAN_DC1_CPUSET=${INFINISPAN_DC1_CPUSETS_ARRAY[0]}
+                INFINISPAN_DC2_CPUSET=${INFINISPAN_DC2_CPUSETS_ARRAY[0]}
+                echo "INFINISPAN_DC1_CPUSET: $INFINISPAN_DC1_CPUSET"
+                echo "INFINISPAN_DC2_CPUSET: $INFINISPAN_DC2_CPUSET"
+
+                validateNotEmpty KEYCLOAK_DC1_CPUSETS $KEYCLOAK_DC1_CPUSETS
+                validateNotEmpty KEYCLOAK_DC2_CPUSETS $KEYCLOAK_DC2_CPUSETS
+                KEYCLOAK_DC1_CPUSETS_ARRAY=( $KEYCLOAK_DC1_CPUSETS )
+                KEYCLOAK_DC2_CPUSETS_ARRAY=( $KEYCLOAK_DC2_CPUSETS )
+                KEYCLOAK_DC1_MAX_SCALE=${#KEYCLOAK_DC1_CPUSETS_ARRAY[@]}
+                KEYCLOAK_DC2_MAX_SCALE=${#KEYCLOAK_DC2_CPUSETS_ARRAY[@]}
+                KEYCLOAK_DC1_SCALE="${KEYCLOAK_DC1_SCALE:-$KEYCLOAK_DC1_MAX_SCALE}"
+                KEYCLOAK_DC2_SCALE="${KEYCLOAK_DC2_SCALE:-$KEYCLOAK_DC2_MAX_SCALE}"
+
+                if [ $KEYCLOAK_DC1_SCALE -gt $KEYCLOAK_DC1_MAX_SCALE ]; then KEYCLOAK_DC1_SCALE=$KEYCLOAK_DC1_MAX_SCALE; fi
+                if [ $KEYCLOAK_DC1_SCALE -lt 1 ]; then KEYCLOAK_DC1_SCALE=1; fi
+                if [ $KEYCLOAK_DC2_SCALE -gt $KEYCLOAK_DC2_MAX_SCALE ]; then KEYCLOAK_DC2_SCALE=$KEYCLOAK_DC2_MAX_SCALE; fi
+                if [ $KEYCLOAK_DC2_SCALE -lt 1 ]; then KEYCLOAK_DC2_SCALE=1; fi
+
+                echo "KEYCLOAK_DC1_CPUSETS: ${KEYCLOAK_DC1_CPUSETS}"
+                echo "KEYCLOAK_DC2_CPUSETS: ${KEYCLOAK_DC2_CPUSETS}"
+                echo "KEYCLOAK_DC1_SCALE: ${KEYCLOAK_DC1_SCALE} (max ${KEYCLOAK_DC1_MAX_SCALE})"
+                echo "KEYCLOAK_DC2_SCALE: ${KEYCLOAK_DC2_SCALE} (max ${KEYCLOAK_DC2_MAX_SCALE})"
+
+                KEYCLOAK_SERVICES=""
+                STOPPED_KEYCLOAK_SERVICES=""
+                for ((i=1; i<=$KEYCLOAK_DC1_MAX_SCALE; i++)) ; do
+                    if (( $i <= $KEYCLOAK_DC1_SCALE )) ; then
+                        KEYCLOAK_SERVICES="$KEYCLOAK_SERVICES keycloak_dc1_$i"
+                    else
+                        STOPPED_KEYCLOAK_SERVICES="$STOPPED_KEYCLOAK_SERVICES keycloak_dc1_$i"
+                    fi
+                done
+                for ((i=1; i<=$KEYCLOAK_DC2_MAX_SCALE; i++)) ; do
+                    if (( $i <= $KEYCLOAK_DC2_SCALE )) ; then
+                        KEYCLOAK_SERVICES="$KEYCLOAK_SERVICES keycloak_dc2_$i"
+                    else
+                        STOPPED_KEYCLOAK_SERVICES="$STOPPED_KEYCLOAK_SERVICES keycloak_dc2_$i"
+                    fi
+                done
+                echo "BASE_SERVICES: $BASE_SERVICES"
+                echo "KEYCLOAK_SERVICES: $KEYCLOAK_SERVICES"
+
+                generateDockerComposeFile
+
+            ;;
+
+        esac
+
+        runCommand "docker-compose -f $DOCKER_COMPOSE_FILE -p ${PROJECT_NAME} up -d --build $BASE_SERVICES $KEYCLOAK_SERVICES"
+        if [ ! -z "$STOPPED_KEYCLOAK_SERVICES" ] ; then 
+            echo "STOPPED_KEYCLOAK_SERVICES: $STOPPED_KEYCLOAK_SERVICES"
+            runCommand "docker-compose -f $DOCKER_COMPOSE_FILE -p ${PROJECT_NAME} stop $STOPPED_KEYCLOAK_SERVICES"
+        fi
+
+        if isTestDeployment ; then 
+            generateProvisionedSystemProperties; 
+            $PROJECT_BASEDIR/healthcheck.sh
+        fi
+
+    ;;
+
+
+    teardown)
+
+        DELETE_DATA="${DELETE_DATA:-true}"
+        echo "DELETE_DATA: $DELETE_DATA"
+        if "$DELETE_DATA" ; then VOLUMES_ARG="-v"; else VOLUMES_ARG=""; fi
+
+        runCommand "docker-compose -f $DOCKER_COMPOSE_FILE -p ${PROJECT_NAME} down $VOLUMES_ARG"
+
+        if isTestDeployment ; then removeProvisionedSystemProperties; fi
+    ;;
+
+
+    export-dump|import-dump)
+        
+        loadProvisionedSystemProperties
+
+        echo "KEYCLOAK_SERVICES: $KEYCLOAK_SERVICES"
+        if [ -z "$KEYCLOAK_SERVICES" ]; then echo "Unable to load KEYCLOAK_SERVICES"; exit 1; fi
+
+        case "$DEPLOYMENT" in
+            singlenode|cluster) export DB_CONTAINER=${PROJECT_NAME}_mariadb_1 ;;
+            crossdc) export DB_CONTAINER=${PROJECT_NAME}_mariadb_dc1_1 ;;
+            *) echo "Deployment '$DEPLOYMENT' doesn't support operation '$OPERATION'." ; exit 1 ;;
+        esac
+        if [ -z "$DATASET" ]; then echo "Operation '$OPERATION' requires DATASET parameter."; exit 1; fi
+        echo "DATASET: $DATASET"
+
+        echo "Stopping Keycloak services."
+        runCommand "docker-compose -f $DOCKER_COMPOSE_FILE -p ${PROJECT_NAME} stop $KEYCLOAK_SERVICES"
+
+        cd $PROJECT_BASEDIR/datasets
+        case "$OPERATION" in
+            export-dump)
+                echo "Exporting $DATASET.sql."
+                if docker exec $DB_CONTAINER /usr/bin/mysqldump -u root --password=root keycloak > $DATASET.sql ; then 
+                    echo "Compressing $DATASET.sql."
+                    gzip $DATASET.sql
+                fi
+            ;;
+            import-dump) 
+                DUMP_DOWNLOAD_SITE=${DUMP_DOWNLOAD_SITE:-https://downloads.jboss.org/keycloak-qe}
+                if [ ! -f "$DATASET.sql.gz" ]; then 
+                    echo "Downloading dump file."
+                    if ! curl -f -O $DUMP_DOWNLOAD_SITE/$DATASET.properties -O $DUMP_DOWNLOAD_SITE/$DATASET.sql.gz ; then
+                        echo Download failed.
+                        exit 1
+                    fi
+                fi
+                echo "Importing $DATASET.sql.gz"
+                set -o pipefail
+                if ! zcat $DATASET.sql.gz | docker exec -i $DB_CONTAINER /usr/bin/mysql -u root --password=root keycloak ; then
+                    echo Import failed.
+                    exit 1
+                fi
+            ;;
+        esac
+        cd $PROJECT_BUILD_DIRECTORY/docker-compose
+
+        echo "Starting Keycloak services."
+        runCommand "docker-compose -f $DOCKER_COMPOSE_FILE -p ${PROJECT_NAME} up -d --no-recreate $KEYCLOAK_SERVICES"
+        # need to update mapped ports
+        generateProvisionedSystemProperties
+
+        $PROJECT_BASEDIR/healthcheck.sh
+
+    ;;
+
+    *)
+        echo "Unsupported operation: '$OPERATION'"
+        exit 1
+    ;;
+
+esac
+

testsuite/performance/tests/healthcheck.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+cd "$(dirname "$0")"
+. ./common.sh
+
+CHECK_TIMEOUT=${CHECK_TIMEOUT:-5}
+function isKeycloakServerReady {
+    CODE=`curl --connect-timeout $CHECK_TIMEOUT -m $CHECK_TIMEOUT -s -o /dev/null -w "%{http_code}" $1/realms/master`
+    [ "$CODE" -eq "200" ]
+}
+
+
+if [ -f "$PROVISIONED_SYSTEM_PROPERTIES_FILE" ] ; then 
+
+    FRONTEND_SERVERS=$( grep -Po "(?<=^keycloak.frontend.servers=).*" "$PROVISIONED_SYSTEM_PROPERTIES_FILE" )
+    BACKEND_SERVERS=$( grep -Po "(?<=^keycloak.backend.servers=).*" "$PROVISIONED_SYSTEM_PROPERTIES_FILE" )
+    KEYCLOAK_SERVERS="$FRONTEND_SERVERS $BACKEND_SERVERS"
+
+    HEALTHCHECK_ITERATIONS=${HEALTHCHECK_ITERATIONS:-20}
+    HEALTHCHECK_WAIT=${HEALTHCHECK_WAIT:-6s}
+    
+    echo "Waiting for Keycloak servers to be ready."
+    echo "Check intervals: $HEALTHCHECK_ITERATIONS x $HEALTHCHECK_WAIT."
+
+    C=0
+    READY=false
+    while ! $READY; do 
+        C=$((C+1))
+        if [ $C -gt $HEALTHCHECK_ITERATIONS ]; then
+            echo System healthcheck failed.
+            exit 1
+        fi
+        echo $( date -Iseconds )
+        READY=true
+        for SERVER in $KEYCLOAK_SERVERS ; do
+            if isKeycloakServerReady $SERVER; then 
+                echo -e "Keycloak server: $SERVER\tREADY"
+            else
+                echo -e "Keycloak server: $SERVER\tNOT READY"
+                READY=false
+            fi
+        done
+        if ! $READY; then sleep $HEALTHCHECK_WAIT ; fi
+    done
+    echo All servers ready.
+
+else
+    echo Healthcheck skipped.
+fi

testsuite/performance/tests/log-tool.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+cd "$(dirname "$0")"
+. ./common.sh
+
+java -cp $PROJECT_BUILD_DIRECTORY/classes org.keycloak.performance.log.LogProcessor "$@"

testsuite/performance/tests/pom.xml

@@ -29,19 +29,23 @@
     <artifactId>performance-tests</artifactId>
     <name>Keycloak Performance TestSuite - Tests</name>
 
-    <description>
-    </description>
-
     <properties>
-        <compose.file>docker-compose.yml</compose.file>
+        <provisioner>docker-compose</provisioner>
-        <compose.up.params/>
+        <deployment>singlenode</deployment>
-        <compose.restart.params>keycloak</compose.restart.params>
-        <keycloak.server.uris>http://localhost:8080/auth</keycloak.server.uris>
-        <db.url>jdbc:mariadb://keycloak:keycloak@localhost:3306/keycloak</db.url>
         
+        <provisioned.system.properties.file>${project.build.directory}/provisioned-system.properties</provisioned.system.properties.file>
+
+        <!-- Keycloak Server Settings -->
+        <keycloak.scale/>
+        <keycloak.dc1.scale/>
+        <keycloak.dc2.scale/>
+        <keycloak.docker.cpusets>2-3</keycloak.docker.cpusets>
+        <keycloak.dc1.docker.cpusets>2</keycloak.dc1.docker.cpusets>
+        <keycloak.dc2.docker.cpusets>3</keycloak.dc2.docker.cpusets>
+        <keycloak.docker.memlimit>2500m</keycloak.docker.memlimit>
         <keycloak.jvm.memory>-Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</keycloak.jvm.memory>
-        <keycloak.http.max-connections>500</keycloak.http.max-connections>
+        <keycloak.http.max-connections>50000</keycloak.http.max-connections>
-        <keycloak.ajp.max-connections>500</keycloak.ajp.max-connections>
+        <keycloak.ajp.max-connections>50000</keycloak.ajp.max-connections>
         <keycloak.worker.io-threads>2</keycloak.worker.io-threads>
         <keycloak.worker.task-max-threads>16</keycloak.worker.task-max-threads>
         <keycloak.ds.min-pool-size>10</keycloak.ds.min-pool-size>
@@ -49,24 +53,32 @@
         <keycloak.ds.pool-prefill>true</keycloak.ds.pool-prefill>
         <keycloak.ds.ps-cache-size>100</keycloak.ds.ps-cache-size>
 
-        <keycloak-lb.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</keycloak-lb.jvm.memory>
+        <!-- Database Settings -->
-        <keycloak-lb.http.max-connections>500</keycloak-lb.http.max-connections>
+        <db.docker.cpusets>1</db.docker.cpusets>
-        <keycloak-lb.worker.io-threads>2</keycloak-lb.worker.io-threads>
+        <db.dc1.docker.cpusets>1</db.dc1.docker.cpusets>
-        <keycloak-lb.worker.task-max-threads>16</keycloak-lb.worker.task-max-threads>
+        <db.dc2.docker.cpusets>1</db.dc2.docker.cpusets>
-
-        <!-- Docker-related properties -->
-        <db.docker.cpuset>1</db.docker.cpuset>
-        <keycloak.docker.cpuset>2-3</keycloak.docker.cpuset>
-        <keycloak.dc1.docker.cpuset>2</keycloak.dc1.docker.cpuset>
-        <keycloak.dc2.docker.cpuset>3</keycloak.dc2.docker.cpuset>
-        <monitoring.docker.cpuset>0</monitoring.docker.cpuset>
-
         <db.docker.memlimit>2g</db.docker.memlimit>
-        <keycloak.docker.memlimit>2g</keycloak.docker.memlimit>
-        <!-- End of docker-related properties -->
 
-        <infinispan.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC</infinispan.jvm.memory>
+        <!-- Load Balancer Settings -->
+        <lb.docker.cpusets>1</lb.docker.cpusets>
+        <lb.dc1.docker.cpusets>1</lb.dc1.docker.cpusets>
+        <lb.dc2.docker.cpusets>1</lb.dc2.docker.cpusets>
+        <lb.docker.memlimit>1g</lb.docker.memlimit>
+        <lb.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</lb.jvm.memory>
+        <lb.http.max-connections>500</lb.http.max-connections>
+        <lb.worker.io-threads>2</lb.worker.io-threads>
+        <lb.worker.task-max-threads>16</lb.worker.task-max-threads>
 
+        <!-- Infinispan Settings -->
+        <infinispan.dc1.docker.cpusets>1</infinispan.dc1.docker.cpusets>
+        <infinispan.dc2.docker.cpusets>1</infinispan.dc2.docker.cpusets>
+        <infinispan.docker.memlimit>1500m</infinispan.docker.memlimit>
+        <infinispan.jvm.memory>-Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC</infinispan.jvm.memory>
+        
+        <!-- Monitoring Settings -->
+        <monitoring.docker.cpusets>0</monitoring.docker.cpusets>
+
+        <!-- Other -->
         <dataset>default</dataset>
         <numOfWorkers>1</numOfWorkers>
 
@@ -156,6 +168,58 @@
             </testResource>
         </testResources>
         <plugins>
+
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-enforcer-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>enforce-teardown-before-clean</id>
+                        <goals>
+                            <goal>enforce</goal>
+                        </goals>
+                        <phase>pre-clean</phase>
+                        <configuration>
+                            <rules>
+                                <requireFilesDontExist>
+                                    <message><![CDATA[
+                                        WARNING: A previously provisioned system still appears to be running.
+         Please tear it down with `mvn verify -P teardown [-Pcluster|crossdc]` before runing `mvn clean`, 
+         or delete the `provisioned-system.properties` file manually.
+                                    ]]></message>
+                                    <files>
+                                        <file>${provisioned.system.properties.file}</file>
+                                    </files>
+                                </requireFilesDontExist>
+                            </rules>
+                            <fail>true</fail>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+            
+            
+            <plugin>
+                <groupId>org.codehaus.mojo</groupId>
+                <artifactId>properties-maven-plugin</artifactId>
+                <version>1.0.0</version>
+                <executions>            
+                    <execution>
+                        <id>read-existing-provisioned-system-properties</id>
+                        <phase>initialize</phase>
+                        <goals>
+                            <goal>read-project-properties</goal>
+                        </goals>
+                        <configuration>
+                            <files>
+                                <file>${project.build.directory}/provisioned-system.properties</file>
+                            </files>
+                            <quiet>true</quiet>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+            
             <plugin>
                 <groupId>net.alchim31.maven</groupId>
                 <artifactId>scala-maven-plugin</artifactId>
@@ -191,12 +255,11 @@
                 </executions>
             </plugin>
 
-
             <plugin>
                 <!--
                 Execute test directly by using:
 
-                    mvn gatling:execute -f testsuite/performance/gatling -Dgatling.simulationClass=keycloak.DemoSimulation2
+                mvn gatling:execute -Ptest -f testsuite/performance/gatling -Dgatling.simulationClass=keycloak.DemoSimulation2
 
                 For more usage info see: http://gatling.io/docs/current/extensions/maven_plugin/
                 -->
@@ -213,6 +276,7 @@
                         <include>keycloak.DemoSimulation2</include>
                     </includes-->
                     <jvmArgs>
+                        <param>-Dkeycloak.server.uris=${keycloak.frontend.servers}</param>
                         <param>-DnumOfRealms=${numOfRealms}</param>
                         <param>-DusersPerRealm=${usersPerRealm}</param>
                         <param>-DclientsPerRealm=${clientsPerRealm}</param>
@@ -234,20 +298,153 @@
                 </executions>
             </plugin>
 
+            <plugin>
+                <groupId>org.codehaus.mojo</groupId>
+                <artifactId>exec-maven-plugin</artifactId>
+                <configuration>
+                    <workingDirectory>${project.basedir}</workingDirectory>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+    <profiles>
+        
+        <profile>
+            <id>docker-compose</id>
+            <activation>
+                <property>
+                    <name>!provisioner</name>
+                </property>
+            </activation>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-antrun-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>copy-dockerfiles-etc</id>
+                                <phase>generate-resources</phase>
+                                <goals>
+                                    <goal>run</goal>
+                                </goals>
+                                <configuration>
+                                    <target>
+                                        <copy todir="${project.build.directory}/docker-compose" overwrite="false" >
+                                            <fileset dir="${project.basedir}/src/main/docker-compose"/>
+                                        </copy>
+                                        <copy todir="${project.build.directory}/docker-compose" overwrite="false" >
+                                            <fileset dir="${project.basedir}/..">
+                                                <include name="db/**"/>
+                                                <include name="infinispan/**"/>
+                                                <include name="load-balancer/**"/>
+                                                <include name="monitoring/**"/>
+                                            </fileset>
+                                        </copy>
+                                        <copy todir="${project.build.directory}/docker-compose/keycloak" overwrite="false" >
+                                            <fileset dir="${project.basedir}/../keycloak/target/docker"/>
+                                        </copy>
+                                    </target>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
+        
+        <profile>
+            <id>cluster</id>
+            <properties>
+                <deployment>cluster</deployment>
+                <keycloak.docker.cpusets>2 3</keycloak.docker.cpusets>
+            </properties>
+        </profile>
+        <profile>
+            <id>crossdc</id>
+            <properties>
+                <deployment>crossdc</deployment>
+            </properties>
+        </profile>
+        
+        <profile>
+            <id>provision</id>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.codehaus.mojo</groupId>
+                        <artifactId>exec-maven-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>provision</id>
+                                <phase>process-test-resources</phase>
+                                <goals>
+                                    <goal>exec</goal>
+                                </goals>
+                                <configuration>
+                                    <executable>./${provisioner}.sh</executable>
+                                    <environmentVariables>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>${deployment}</DEPLOYMENT>
+                                        <OPERATION>provision</OPERATION>
+
+                                        <KEYCLOAK_VERSION>${project.version}</KEYCLOAK_VERSION>
+                                        
+                                        <KEYCLOAK_SCALE>${keycloak.scale}</KEYCLOAK_SCALE>
+                                        <KEYCLOAK_DC1_SCALE>${keycloak.dc1.scale}</KEYCLOAK_DC1_SCALE>
+                                        <KEYCLOAK_DC2_SCALE>${keycloak.dc2.scale}</KEYCLOAK_DC2_SCALE>
+                                        <KEYCLOAK_CPUSETS>${keycloak.docker.cpusets}</KEYCLOAK_CPUSETS>
+                                        <KEYCLOAK_DC1_CPUSETS>${keycloak.dc1.docker.cpusets}</KEYCLOAK_DC1_CPUSETS>
+                                        <KEYCLOAK_DC2_CPUSETS>${keycloak.dc2.docker.cpusets}</KEYCLOAK_DC2_CPUSETS>
+                                        <KEYCLOAK_MEMLIMIT>${keycloak.docker.memlimit}</KEYCLOAK_MEMLIMIT>
+                                        <KEYCLOAK_JVM_MEMORY>${keycloak.jvm.memory}</KEYCLOAK_JVM_MEMORY>
+                                        <KEYCLOAK_HTTP_MAX_CONNECTIONS>${keycloak.http.max-connections}</KEYCLOAK_HTTP_MAX_CONNECTIONS>
+                                        <KEYCLOAK_AJP_MAX_CONNECTIONS>${keycloak.ajp.max-connections}</KEYCLOAK_AJP_MAX_CONNECTIONS>
+                                        <KEYCLOAK_WORKER_IO_THREADS>${keycloak.worker.io-threads}</KEYCLOAK_WORKER_IO_THREADS>
+                                        <KEYCLOAK_WORKER_TASK_MAX_THREADS>${keycloak.worker.task-max-threads}</KEYCLOAK_WORKER_TASK_MAX_THREADS>
+                                        <KEYCLOAK_DS_MIN_POOL_SIZE>${keycloak.ds.min-pool-size}</KEYCLOAK_DS_MIN_POOL_SIZE>
+                                        <KEYCLOAK_DS_MAX_POOL_SIZE>${keycloak.ds.max-pool-size}</KEYCLOAK_DS_MAX_POOL_SIZE>
+                                        <KEYCLOAK_DS_POOL_PREFILL>${keycloak.ds.pool-prefill}</KEYCLOAK_DS_POOL_PREFILL>
+                                        <KEYCLOAK_DS_PS_CACHE_SIZE>${keycloak.ds.ps-cache-size}</KEYCLOAK_DS_PS_CACHE_SIZE>
+
+                                        <DB_CPUSETS>${db.docker.cpusets}</DB_CPUSETS>
+                                        <DB_DC1_CPUSETS>${db.dc1.docker.cpusets}</DB_DC1_CPUSETS>
+                                        <DB_DC2_CPUSETS>${db.dc2.docker.cpusets}</DB_DC2_CPUSETS>
+                                        <DB_MEMLIMIT>${db.docker.memlimit}</DB_MEMLIMIT>
+
+                                        <LB_CPUSETS>${lb.docker.cpusets}</LB_CPUSETS>
+                                        <LB_DC1_CPUSETS>${lb.dc1.docker.cpusets}</LB_DC1_CPUSETS>
+                                        <LB_DC2_CPUSETS>${lb.dc2.docker.cpusets}</LB_DC2_CPUSETS>
+                                        <LB_MEMLIMIT>${lb.docker.memlimit}</LB_MEMLIMIT>
+                                        <LB_JVM_MEMORY>${lb.jvm.memory}</LB_JVM_MEMORY>
+                                        <LB_HTTP_MAX_CONNECTIONS>${lb.http.max-connections}</LB_HTTP_MAX_CONNECTIONS>
+                                        <LB_WORKER_IO_THREADS>${lb.worker.io-threads}</LB_WORKER_IO_THREADS>
+                                        <LB_WORKER_TASK_MAX_THREADS>${lb.worker.task-max-threads}</LB_WORKER_TASK_MAX_THREADS>
+                                        
+                                        <INFINISPAN_DC1_CPUSETS>${infinispan.dc1.docker.cpusets}</INFINISPAN_DC1_CPUSETS>
+                                        <INFINISPAN_DC2_CPUSETS>${infinispan.dc2.docker.cpusets}</INFINISPAN_DC2_CPUSETS>
+                                        <INFINISPAN_MEMLIMIT>${infinispan.docker.memlimit}</INFINISPAN_MEMLIMIT>
+                                        <INFINISPAN_JVM_MEMORY>${infinispan.jvm.memory}</INFINISPAN_JVM_MEMORY>
+                                    </environmentVariables>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
                     <plugin>
                         <groupId>org.codehaus.mojo</groupId>
                         <artifactId>properties-maven-plugin</artifactId>
                         <version>1.0.0</version>
                         <executions>
                             <execution>
-                        <id>read-dataset-properties</id>
+                                <id>read-new-provisioned-system-properties</id>
-                        <phase>initialize</phase>
                                 <goals>
                                     <goal>read-project-properties</goal>
                                 </goals>
+                                <phase>pre-integration-test</phase>
                                 <configuration>
                                     <files>
-                                <file>${project.basedir}/datasets/${dataset}.properties</file>
+                                        <file>${project.build.directory}/provisioned-system.properties</file>
                                     </files>
                                 </configuration>
                             </execution>
@@ -255,8 +452,8 @@
                     </plugin>
                 </plugins>
             </build>
+        </profile>
         
-    <profiles>
         <profile>
             <id>initialize-dataset-properties</id>
             <activation>
@@ -273,123 +470,15 @@
                         <executions>
                             <execution>
                                 <id>initialize-dataset-properties</id>
-                                <phase>initialize</phase>
+                                <phase>pre-integration-test</phase>
                                 <goals>
                                     <goal>read-project-properties</goal>
                                 </goals>
                                 <configuration>
                                     <files>
-                                        <file>${project.basedir}/datasets/${dataset}.properties</file>
+                                        <file>${project.basedir}/../datasets/${dataset}.properties</file>
                                     </files>
-                                </configuration>
+                                    <quiet>true</quiet>
-                            </execution>
-                        </executions>
-                    </plugin>                    
-
-                </plugins>
-            </build>
-        </profile>
-        
-        <profile>
-            <id>cluster</id>
-            <properties>
-                <compose.file>docker-compose-cluster.yml</compose.file>
-                <keycloak.scale>1</keycloak.scale>
-                <compose.up.params>--scale keycloak=${keycloak.scale}</compose.up.params>
-                <keycloak.server.uris>http://localhost:8080/auth</keycloak.server.uris>
-            </properties>
-        </profile>
-        <profile>
-            <id>crossdc</id>
-            <properties>
-                <compose.file>docker-compose-crossdc.yml</compose.file>
-                <keycloak.dc1.scale>1</keycloak.dc1.scale>
-                <keycloak.dc2.scale>1</keycloak.dc2.scale>
-                <compose.up.params>--scale keycloak_dc1=${keycloak.dc1.scale} --scale keycloak_dc2=${keycloak.dc2.scale}</compose.up.params>
-                <compose.restart.params>keycloak_dc1 keycloak_dc2</compose.restart.params>
-                <keycloak.server.uris>http://localhost:8081/auth http://localhost:8082/auth</keycloak.server.uris>
-            </properties>
-        </profile>
-        
-        
-        <profile>
-            <id>provision</id>
-            <build>
-                <plugins>
-                    <plugin>
-                        <groupId>org.codehaus.mojo</groupId>
-                        <artifactId>exec-maven-plugin</artifactId>
-                        <executions>
-                            <execution>
-                                <id>docker-compose-up</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>exec</goal>
-                                </goals>
-                                <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
-                                    <executable>docker-compose</executable>
-                                    <commandlineArgs>-f ${compose.file} up -d --build ${compose.up.params}</commandlineArgs>
-                                    <environmentVariables>
-                                        <KEYCLOAK_VERSION>${project.version}</KEYCLOAK_VERSION>
-
-                                        <KEYCLOAK_CPUSET>${keycloak.docker.cpuset}</KEYCLOAK_CPUSET>
-                                        <KEYCLOAK_DC1_CPUSET>${keycloak.dc1.docker.cpuset}</KEYCLOAK_DC1_CPUSET>
-                                        <KEYCLOAK_DC2_CPUSET>${keycloak.dc2.docker.cpuset}</KEYCLOAK_DC2_CPUSET>
-                                        <KEYCLOAK_MEMLIMIT>${keycloak.docker.memlimit}</KEYCLOAK_MEMLIMIT>
-                                        <DB_CPUSET>${db.docker.cpuset}</DB_CPUSET>
-                                        <DB_MEMLIMIT>${db.docker.memlimit}</DB_MEMLIMIT>
-                                        
-                                        <KEYCLOAK_JVM_MEMORY>${keycloak.jvm.memory}</KEYCLOAK_JVM_MEMORY>
-                                        <KEYCLOAK_HTTP_MAX_CONNECTIONS>${keycloak.http.max-connections}</KEYCLOAK_HTTP_MAX_CONNECTIONS>
-                                        <KEYCLOAK_AJP_MAX_CONNECTIONS>${keycloak.ajp.max-connections}</KEYCLOAK_AJP_MAX_CONNECTIONS>
-                                        <KEYCLOAK_WORKER_IO_THREADS>${keycloak.worker.io-threads}</KEYCLOAK_WORKER_IO_THREADS>
-                                        <KEYCLOAK_WORKER_TASK_MAX_THREADS>${keycloak.worker.task-max-threads}</KEYCLOAK_WORKER_TASK_MAX_THREADS>
-                                        <KEYCLOAK_DS_MIN_POOL_SIZE>${keycloak.ds.min-pool-size}</KEYCLOAK_DS_MIN_POOL_SIZE>
-                                        <KEYCLOAK_DS_MAX_POOL_SIZE>${keycloak.ds.max-pool-size}</KEYCLOAK_DS_MAX_POOL_SIZE>
-                                        <KEYCLOAK_DS_POOL_PREFILL>${keycloak.ds.pool-prefill}</KEYCLOAK_DS_POOL_PREFILL>
-                                        <KEYCLOAK_DS_PS_CACHE_SIZE>${keycloak.ds.ps-cache-size}</KEYCLOAK_DS_PS_CACHE_SIZE>
-                                        
-                                        <KEYCLOAK_LB_JVM_MEMORY>${keycloak-lb.jvm.memory}</KEYCLOAK_LB_JVM_MEMORY>
-                                        <KEYCLOAK_LB_HTTP_MAX_CONNECTIONS>${keycloak-lb.http.max-connections}</KEYCLOAK_LB_HTTP_MAX_CONNECTIONS>
-                                        <KEYCLOAK_LB_WORKER_IO_THREADS>${keycloak-lb.worker.io-threads}</KEYCLOAK_LB_WORKER_IO_THREADS>
-                                        <KEYCLOAK_LB_WORKER_TASK_MAX_THREADS>${keycloak-lb.worker.task-max-threads}</KEYCLOAK_LB_WORKER_TASK_MAX_THREADS>
-                                        
-                                        <INFINISPAN_JVM_MEMORY>${infinispan.jvm.memory}</INFINISPAN_JVM_MEMORY>
-                                    </environmentVariables>
-                                </configuration>
-                            </execution>
-                            <execution>
-                                <id>healthcheck</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>exec</goal>
-                                </goals>
-                                <configuration>
-                                    <executable>./healthcheck.sh</executable>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
-                                    <environmentVariables>
-                                        <KEYCLOAK_SERVER_URIS>${keycloak.server.uris}</KEYCLOAK_SERVER_URIS>
-                                    </environmentVariables>
-                                </configuration>
-                            </execution>
-                        </executions>
-                    </plugin>
-                    <plugin>
-                        <artifactId>maven-antrun-plugin</artifactId>
-                        <executions>
-                            <execution>
-                                <id>write-provisioned-system-properties</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>run</goal>
-                                </goals>
-                                <configuration>
-                                    <target>
-                                        <propertyfile file="${project.build.directory}/provisioned-system.properties">
-                                            <entry key="keycloak.server.uris" value="${keycloak.server.uris}"/>
-                                        </propertyfile>
-                                    </target>
                                 </configuration>
                             </execution>
                         </executions>
@@ -442,6 +531,7 @@
                                     <arguments>
                                         <argument>-classpath</argument>
                                         <classpath/>
+                                        <argument>-Dkeycloak.server.uris=${keycloak.frontend.servers}</argument>
                                         <argument>-DnumOfWorkers=${numOfWorkers}</argument>
                                         <argument>org.keycloak.performance.RealmsConfigurationLoader</argument>
                                         <argument>benchmark-realms.json</argument>
@@ -450,48 +540,34 @@
                             </execution>
                         </executions>
                     </plugin>
-                    <plugin>
-                        <artifactId>maven-antrun-plugin</artifactId>
-                        <executions>
-                            <execution>
-                                <id>write-imported-dataset-properties</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>run</goal>
-                                </goals>
-                                <configuration>
-                                    <target>
-                                        <propertyfile file="${project.build.directory}/imported-dataset.properties">
-                                            <entry key="numOfRealms" value="${numOfRealms}"/>
-                                            <entry key="usersPerRealm" value="${usersPerRealm}"/>
-                                            <entry key="clientsPerRealm" value="${clientsPerRealm}"/>
-                                            <entry key="realmRoles" value="${realmRoles}"/>
-                                            <entry key="realmRolesPerUser" value="${realmRolesPerUser}"/>
-                                            <entry key="clientRolesPerUser" value="${clientRolesPerUser}"/>
-                                            <entry key="clientRolesPerClient" value="${clientRolesPerClient}"/>
-                                            <entry key="hashIterations" value="${hashIterations}"/>
-                                        </propertyfile>
-                                    </target>
-                                </configuration>
-                            </execution>
-                        </executions>
-                    </plugin>
                 </plugins>
             </build>
         </profile>
 
         <profile>
-            <id>export-dump-after-generation</id>
+            <id>export-dump</id>
-
-            <activation>
-                <activeByDefault>false</activeByDefault>
-                <property>
-                    <name>export-dump</name>
-                </property>
-            </activation>
-
             <build>
                 <plugins>
+                    <plugin>
+                        <artifactId>maven-enforcer-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>enforce-nondefault-dataset</id>
+                                <goals>
+                                    <goal>enforce</goal>
+                                </goals>
+                                <configuration>
+                                    <rules>
+                                        <requireProperty>
+                                            <property>dataset</property>
+                                            <regex>(?!default).*</regex>
+                                            <regexMessage>For the "export-dump" task property "dataset" cannot be set to "default".</regexMessage>
+                                        </requireProperty>
+                                    </rules>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
                     <plugin>
                         <groupId>org.codehaus.mojo</groupId>
                         <artifactId>exec-maven-plugin</artifactId>
@@ -503,10 +579,11 @@
                                     <goal>exec</goal>
                                 </goals>
                                 <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./${provisioner}.sh</executable>
-                                    <executable>./prepare-dump.sh</executable>
-
                                     <environmentVariables>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>${deployment}</DEPLOYMENT>
+                                        <OPERATION>export-dump</OPERATION>
                                         <DATASET>${dataset}</DATASET>
                                     </environmentVariables>
                                 </configuration>
@@ -526,46 +603,21 @@
                         <artifactId>exec-maven-plugin</artifactId>
                         <executions>
                             <execution>
-                                <id>load-dump</id>
+                                <id>import-dump</id>
                                 <phase>pre-integration-test</phase>
                                 <goals>
                                     <goal>exec</goal>
                                 </goals>
                                 <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./${provisioner}.sh</executable>
-                                    <executable>./load-dump.sh</executable>
-
                                     <environmentVariables>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>${deployment}</DEPLOYMENT>
+                                        <OPERATION>import-dump</OPERATION>
                                         <DATASET>${dataset}</DATASET>
                                     </environmentVariables>
                                 </configuration>
                             </execution>
-                            <execution>
-                                <id>restart-keycloak</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>exec</goal>
-                                </goals>
-                                <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
-                                    <executable>docker-compose</executable>
-                                    <commandlineArgs>-f ${compose.file} restart ${compose.restart.params}</commandlineArgs>
-                                </configuration>
-                            </execution>
-                            <execution>
-                                <id>healthcheck</id>
-                                <phase>pre-integration-test</phase>
-                                <goals>
-                                    <goal>exec</goal>
-                                </goals>
-                                <configuration>
-                                    <executable>./healthcheck.sh</executable>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
-                                    <environmentVariables>
-                                        <KEYCLOAK_SERVER_URIS>${keycloak.server.uris}</KEYCLOAK_SERVER_URIS>
-                                    </environmentVariables>
-                                </configuration>
-                            </execution>
                         </executions>
                     </plugin>
                 </plugins>
@@ -582,7 +634,7 @@
         <profile>
             <id>teardown</id>
             <properties>
-                <volumes.arg>-v</volumes.arg>
+                <delete.data>true</delete.data>
             </properties>
             <build>
                 <plugins>
@@ -591,17 +643,18 @@
                         <artifactId>exec-maven-plugin</artifactId>
                         <executions>
                             <execution>
-                                <id>docker-compose-down</id>
+                                <id>teardown</id>
                                 <phase>post-integration-test</phase>
                                 <goals>
                                     <goal>exec</goal>
                                 </goals>
                                 <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./${provisioner}.sh</executable>
-                                    <executable>docker-compose</executable>
-                                    <commandlineArgs>-f ${compose.file} down ${volumes.arg}</commandlineArgs>
                                     <environmentVariables>
-                                        <KEYCLOAK_VERSION>${project.version}</KEYCLOAK_VERSION>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>${deployment}</DEPLOYMENT>
+                                        <OPERATION>teardown</OPERATION>
+                                        <DELETE_DATA>${delete.data}</DELETE_DATA>
                                     </environmentVariables>
                                 </configuration>
                             </execution>
@@ -613,7 +666,7 @@
         <profile>
             <id>keep-data</id>
             <properties>
-                <volumes.arg/>
+                <delete.data>false</delete.data>
             </properties>
         </profile>
         
@@ -627,17 +680,17 @@
                         <artifactId>exec-maven-plugin</artifactId>
                         <executions>
                             <execution>
-                                <id>monitoring-docker-compose-up</id>
+                                <id>monitoring-on</id>
                                 <phase>pre-integration-test</phase>
                                 <goals>
                                     <goal>exec</goal>
                                 </goals>
                                 <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./${provisioner}.sh</executable>
-                                    <executable>docker-compose</executable>
-                                    <commandlineArgs>-f docker-compose-monitoring.yml up -d --build</commandlineArgs>
                                     <environmentVariables>
-                                        <MONITORING_CPUSET>${monitoring.docker.cpuset}</MONITORING_CPUSET>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>monitoring</DEPLOYMENT>
+                                        <OPERATION>provision</OPERATION>
                                     </environmentVariables>
                                 </configuration>
                             </execution>
@@ -649,7 +702,7 @@
         <profile>
             <id>monitoring-off</id>
             <properties>
-                <monitoring.volumes.arg/>
+                <delete.monitoring.data>false</delete.monitoring.data>
             </properties>
             <build>
                 <plugins>
@@ -658,15 +711,19 @@
                         <artifactId>exec-maven-plugin</artifactId>
                         <executions>
                             <execution>
-                                <id>monitoring-docker-compose-down</id>
+                                <id>monitoring-off</id>
                                 <phase>post-integration-test</phase>
                                 <goals>
                                     <goal>exec</goal>
                                 </goals>
                                 <configuration>
-                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./${provisioner}.sh</executable>
-                                    <executable>docker-compose</executable>
+                                    <environmentVariables>
-                                    <commandlineArgs>-f docker-compose-monitoring.yml down ${monitoring.volumes.arg}</commandlineArgs>
+                                        <PROVISIONER>${provisioner}</PROVISIONER>
+                                        <DEPLOYMENT>monitoring</DEPLOYMENT>
+                                        <OPERATION>teardown</OPERATION>
+                                        <DELETE_DATA>${delete.monitoring.data}</DELETE_DATA>
+                                    </environmentVariables>
                                 </configuration>
                             </execution>
                         </executions>
@@ -677,7 +734,7 @@
         <profile>
             <id>delete-monitoring-data</id>
             <properties>
-                <monitoring.volumes.arg>-v</monitoring.volumes.arg>
+                <delete.monitoring.data>true</delete.monitoring.data>
             </properties>
         </profile>
 

testsuite/performance/tests/src/main/docker-compose/cluster/docker-compose-base.yml

@@ -0,0 +1,51 @@
+version: "2.2"
+
+networks:
+    keycloak:
+        ipam:
+            config:
+            - subnet: 10.0.1.0/24
+#    loadbalancing:
+#        ipam:
+#            config:
+#            - subnet: 10.0.2.0/24
+
+services:
+    
+    mariadb:
+        build: db/mariadb
+        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
+        cpuset: ${DB_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-1g}
+        networks:
+            - keycloak
+        environment:
+            MYSQL_ROOT_PASSWORD: root
+            MYSQL_DATABASE: keycloak
+            MYSQL_USER: keycloak
+            MYSQL_PASSWORD: keycloak
+        ports:
+            - "3306:3306"
+
+    loadbalancer:
+        build: load-balancer/wildfly-modcluster
+        image: keycloak_test_loadbalancer:${KEYCLOAK_VERSION:-latest}
+#        depends_on:
+#            keycloak:
+#                condition: service_healthy
+        cpuset: ${LB_CPUSET:-1}
+        mem_limit: ${LB_MEMLIMIT:-1g}
+        networks:
+            - keycloak
+#            - loadbalancing
+        environment:
+            PRIVATE_SUBNET: 10.0.1.0/24
+#            PUBLIC_SUBNET: 10.0.2.0/24
+            JAVA_OPTS: ${LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${LB_HTTP_MAX_CONNECTIONS:-500}
+            WORKER_IO_THREADS: ${LB_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
+        ports:
+            - "8080:8080"
+
+

testsuite/performance/tests/src/main/docker-compose/cluster/docker-compose-keycloak.yml

@@ -0,0 +1,34 @@
+    keycloak_%I%:
+        build: keycloak
+        image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
+        depends_on:
+            mariadb:
+                condition: service_healthy
+        cpuset: "%CPUSET%"
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
+        networks:
+            - keycloak
+        environment:
+            CONFIGURATION: standalone-ha.xml
+            PUBLIC_SUBNET: 10.0.1.0/24
+            PRIVATE_SUBNET: 10.0.1.0/24
+            MARIADB_HOSTS: mariadb:3306
+            MARIADB_DATABASE: keycloak
+            MARIADB_USER: keycloak
+            MARIADB_PASSWORD: keycloak
+            KEYCLOAK_USER: admin
+            KEYCLOAK_PASSWORD: admin
+
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-50000}
+            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-50000}
+            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
+            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
+            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
+            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
+            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
+        ports:
+            - "8080"
+            - "9990"
+

testsuite/performance/tests/src/main/docker-compose/crossdc/docker-compose-base.yml

@@ -0,0 +1,154 @@
+version: "2.2"
+
+networks:
+    # DC 1
+    dc1_keycloak:
+        ipam:
+            config:
+            - subnet: 10.1.1.0/24
+    # DC 2
+    dc2_keycloak:
+        ipam:
+            config:
+            - subnet: 10.2.1.0/24
+#    # cross-DC
+#    loadbalancing:
+#        ipam:
+#            config:
+#            - subnet: 10.0.2.0/24
+    # cross-DC
+    db_replication:
+        ipam:
+            config:
+            - subnet: 10.0.3.0/24
+    # cross-DC
+    ispn_replication:
+        ipam:
+            config:
+            - subnet: 10.0.4.0/24
+        
+services:
+
+    infinispan_dc1:
+        build: 
+            context: infinispan
+            args:
+                LOCAL_SITE: dc1
+                REMOTE_SITE: dc2
+        image: keycloak_test_infinispan_dc1:${KEYCLOAK_VERSION:-latest}
+        cpuset: ${INFINISPAN_DC1_CPUSET:-1}
+        mem_limit: ${INFINISPAN_MEMLIMIT:-1500m}
+        networks:
+            - ispn_replication
+            - dc1_keycloak
+        environment:
+            PUBLIC_SUBNET: 10.1.1.0/24
+            PRIVATE_SUBNET: 10.0.4.0/24
+            MGMT_USER: admin
+            MGMT_USER_PASSWORD: admin
+            TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600]
+            JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+        ports:
+            - "9991:9990"
+
+    infinispan_dc2:
+        build: 
+            context: infinispan
+            args:
+                LOCAL_SITE: dc2
+                REMOTE_SITE: dc1
+        image: keycloak_test_infinispan_dc2:${KEYCLOAK_VERSION:-latest}
+        depends_on:
+            infinispan_dc1:
+                condition: service_healthy
+        cpuset: ${INFINISPAN_DC2_CPUSET:-1}
+        mem_limit: ${INFINISPAN_MEMLIMIT:-1500m}
+        networks:
+            - ispn_replication
+            - dc2_keycloak
+        environment:
+            PUBLIC_SUBNET: 10.2.1.0/24
+            PRIVATE_SUBNET: 10.0.4.0/24
+            MGMT_USER: admin
+            MGMT_USER_PASSWORD: admin
+            TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600],infinispan_dc2[7600]
+            JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+        ports:
+            - "9992:9990"
+    
+    
+    mariadb_dc1:
+        build: db/mariadb
+        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
+        cpuset: ${DB_DC1_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-2g}
+        networks:
+            - db_replication
+            - dc1_keycloak
+        environment:
+            MYSQL_ROOT_PASSWORD: root
+            MYSQL_INITDB_SKIP_TZINFO: foo
+            MYSQL_DATABASE: keycloak
+            MYSQL_USER: keycloak
+            MYSQL_PASSWORD: keycloak
+        entrypoint: docker-entrypoint-wsrep.sh
+        command: --wsrep-new-cluster
+        ports:
+            - "3306:3306"
+            
+    mariadb_dc2:
+        build: db/mariadb
+        image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
+        depends_on: 
+            mariadb_dc1:
+                condition: service_healthy
+        cpuset: ${DB_DC2_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-2g}
+        networks:
+            - db_replication
+            - dc2_keycloak
+        environment:
+            MYSQL_ROOT_PASSWORD: root
+            MYSQL_INITDB_SKIP_TZINFO: foo
+        entrypoint: docker-entrypoint-wsrep.sh
+        command: --wsrep_cluster_address=gcomm://mariadb_dc1
+        ports:
+            - "3307:3306"
+
+
+    loadbalancer_dc1:
+        build: load-balancer/wildfly-modcluster
+        image: keycloak_test_loadbalancer:${KEYCLOAK_VERSION:-latest}
+        cpuset: ${LB_DC1_CPUSET:-1}
+        mem_limit: ${LB_MEMLIMIT:-1g}
+        networks:
+            - dc1_keycloak
+#            - loadbalancing
+        environment:
+            PRIVATE_SUBNET: 10.1.1.0/24
+#            PUBLIC_SUBNET: 10.0.2.0/24
+            JAVA_OPTS: ${LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${LB_HTTP_MAX_CONNECTIONS:-500}
+            WORKER_IO_THREADS: ${LB_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
+        ports:
+            - "8081:8080"
+            
+    loadbalancer_dc2:
+        build: load-balancer/wildfly-modcluster
+        image: keycloak_test_loadbalancer:${KEYCLOAK_VERSION:-latest}
+        cpuset: ${LB_DC2_CPUSET:-1}
+        mem_limit: ${LB_MEMLIMIT:-1g}
+        networks:
+            - dc2_keycloak
+#            - loadbalancing
+        environment:
+            PRIVATE_SUBNET: 10.2.1.0/24
+#            PUBLIC_SUBNET: 10.0.2.0/24
+            JAVA_OPTS: ${LB_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${LB_HTTP_MAX_CONNECTIONS:-500}
+            WORKER_IO_THREADS: ${LB_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
+        ports:
+            - "8082:8080"
+

testsuite/performance/tests/src/main/docker-compose/crossdc/docker-compose-keycloak_dc1.yml

@@ -0,0 +1,40 @@
+    keycloak_dc1_%I%:
+        build: ./keycloak
+        image: keycloak_test_keycloak_dc1:${KEYCLOAK_VERSION:-latest}
+        depends_on:
+            # wait for the db cluster to be ready before starting keycloak
+            mariadb_dc2:
+                condition: service_healthy
+            # wait for the ispn cluster to be ready before starting keycloak
+            infinispan_dc2:
+                condition: service_healthy
+        cpuset: "%CPUSET%"
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
+        networks:
+            - dc1_keycloak
+        environment:
+            CONFIGURATION: standalone-ha.xml
+            PUBLIC_SUBNET: 10.1.1.0/24
+            PRIVATE_SUBNET: 10.1.1.0/24
+            MARIADB_HOSTS: mariadb_dc1:3306
+            MARIADB_DATABASE: keycloak
+            MARIADB_USER: keycloak
+            MARIADB_PASSWORD: keycloak
+            KEYCLOAK_USER: admin
+            KEYCLOAK_PASSWORD: admin
+            INFINISPAN_HOST: infinispan_dc1
+            SITE: dc1
+            
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-50000}
+            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-50000}
+            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
+            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
+            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
+            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
+            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
+        ports:
+            - "8080"
+            - "9990"
+

testsuite/performance/tests/src/main/docker-compose/crossdc/docker-compose-keycloak_dc2.yml

@@ -0,0 +1,35 @@
+    keycloak_dc2_%I%:
+        build: ./keycloak
+        image: keycloak_test_keycloak_dc2:${KEYCLOAK_VERSION:-latest}
+        depends_on:
+            # wait for first kc instance to be ready before starting another
+            keycloak_dc1_1:
+                condition: service_healthy
+        cpuset: "%CPUSET%"
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
+        networks:
+            - dc2_keycloak
+        environment:
+            CONFIGURATION: standalone-ha.xml
+            PUBLIC_SUBNET: 10.2.1.0/24
+            PRIVATE_SUBNET: 10.2.1.0/24
+            MARIADB_HOSTS: mariadb_dc2:3306
+            MARIADB_DATABASE: keycloak
+            MARIADB_USER: keycloak
+            MARIADB_PASSWORD: keycloak
+            INFINISPAN_HOST: infinispan_dc2
+            SITE: dc2
+
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-50000}
+            AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-50000}
+            WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
+            WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
+            DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}
+            DS_MAX_POOL_SIZE: ${KEYCLOAK_DS_MAX_POOL_SIZE:-100}
+            DS_POOL_PREFILL: "${KEYCLOAK_DS_POOL_PREFILL:-true}"
+            DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
+        ports:
+            - "8080"
+            - "9990"
+

testsuite/performance/tests/src/main/docker-compose/docker-compose.yml

@@ -12,7 +12,7 @@ services:
         build: db/mariadb
         image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
         cpuset: ${DB_CPUSET:-1}
-        mem_limit: ${DB_MEMLIMIT:-1g}
+        mem_limit: ${DB_MEMLIMIT:-2g}
         networks:
             - keycloak
         environment:
@@ -43,7 +43,7 @@ services:
             KEYCLOAK_PASSWORD: admin
             # docker-compose syntax note: ${ENV_VAR:-<DEFAULT_VALUE>}
             JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
-            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
+            HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-50000}
             WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
             WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
             DS_MIN_POOL_SIZE: ${KEYCLOAK_DS_MIN_POOL_SIZE:-10}

testsuite/performance/tests/src/main/java/org/keycloak/performance/RealmsConfigurationLoader.java

@@ -58,6 +58,7 @@ public class RealmsConfigurationLoader {
     static boolean realmCreated;
 
     public static void main(String [] args) throws IOException {
+        System.out.println("Keycloak servers: "+TestConfig.serverUrisList);
 
         if (args.length == 0) {
             args = new String[] {EXPORT_FILENAME};

testsuite/performance/tests/src/main/java/org/keycloak/performance/TestConfig.java

@@ -4,6 +4,7 @@ import org.keycloak.performance.util.FilteredIterator;
 import org.keycloak.performance.util.LoopingIterator;
 
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Iterator;
 import java.util.List;
 import java.util.concurrent.ConcurrentHashMap;
@@ -77,21 +78,17 @@ public class TestConfig {
 
     static {
         // if KEYCLOAK_SERVER_URIS env var is set, and system property serverUris is not set
-        String servers = System.getProperty("serverUris");
+        String serversProp = System.getProperty("keycloak.server.uris");
-        if (servers == null) {
+        if (serversProp == null) {
-            String env = System.getenv("KEYCLOAK_SERVER_URIS");
+            String serversEnv = System.getenv("KEYCLOAK_SERVERS");
-            serverUris = env != null ? env : "http://localhost:8080/auth";
+            serverUris = serversEnv != null ? serversEnv : "http://localhost:8080/auth";
         } else {
-            serverUris = servers;
+            serverUris = serversProp;
         }
 
         // initialize serverUrisList and serverUrisIterator
-        ArrayList<String> uris = new ArrayList<>();
+        serverUrisList = Arrays.asList(serverUris.split(" "));
-        for (String uri: serverUris.split(" ")) {
+        serverUrisIterator = new LoopingIterator<>(serverUrisList);
-            uris.add(uri);
-        }
-        serverUrisList = uris;
-        serverUrisIterator = new LoopingIterator<>(uris);
     }
 
     // Users iterators by realm

testsuite/performance/tests/src/test/scala/keycloak/AdminConsoleSimulation.scala

@@ -14,7 +14,7 @@ import SimulationsHelper._
 class AdminConsoleSimulation extends Simulation {
 
   println()
-  println("Using server: " + TestConfig.serverUrisList.get(0))
+  println("Target server: " + TestConfig.serverUrisList.get(0))
   println()
   println("Using test parameters:")
   println("  runUsers: " + TestConfig.runUsers)

testsuite/performance/tests/src/test/scala/keycloak/DefaultSimulation.scala

@@ -25,6 +25,9 @@ class DefaultSimulation extends Simulation {
 
 
 
+  println()
+  println("Taget servers: " + TestConfig.serverUrisList)
+  println()
   println("Using test parameters:")
   println("  runUsers: " + TestConfig.runUsers)
   println("  numOfIterations: " + TestConfig.numOfIterations)