Merge pull request #4518 from hmlnarik/KEYCLOAK-5576-Performance-testsuite-should-allow-exporting-the-dump-directly-after-data-generation

KEYCLOAK-5576, KEYCLOAK-5577 Performance testsuite adjustments

testsuite/performance/README.md

@@ -70,7 +70,7 @@ Provisioning/teardown is performed via `docker-compose` tool. More details in [R
 
 ### Generate Test Data
 
-Usage: `mvn verify -Pgenerate-data[,cluster] [-Ddataset=DATASET] [-D<dataset.property>=<value>]`.
+Usage: `mvn verify -Pgenerate-data[,cluster] [-Ddataset=DATASET] [-Dexport-dump] [-D<dataset.property>=<value>]`.
 
 Dataset properties are loaded from `datasets/${dataset}.properties` file. Individual properties can be overriden by specifying `-D` params.
 
@@ -85,8 +85,9 @@ Dataset data is first generated as a .json file, and then imported into Keycloak
 The data can also be exported from the database, and stored locally as `datasets/${dataset}.sql.gz`
 `DATASET=100u ./prepare-dump.sh`
 
-If there is a data dump file available then -Pimport-dump can be used to import the data directly into the database, 
+To speed up dataset initialization part, it is possible to pass `-Dexport-dump` option to have the generated dataset
-by-passing Keycloak server completely.
+exported right after it has been generated. Then, if there is a data dump file available then `-Pimport-dump` 
+can be used to import the data directly into the database, bypassing Keycloak server completely.
 
 Usage: `mvn verify -Pimport-dump [-Ddataset=DATASET]`
 

testsuite/performance/README.provisioning-parameters.md

@@ -4,7 +4,7 @@
 
 | Category    | Setting                       | Property                           | Default value                                                    |
 |-------------|-------------------------------|------------------------------------|------------------------------------------------------------------|
-| JVM         | Memory settings               | `keycloak.jvm.memory`              | -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m |
+| JVM         | Memory settings               | `keycloak.jvm.memory`              | -Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m   |
 | Undertow    | HTTP Listener max connections | `keycloak.http.max-connections`    | 500                                                              |
 |             | AJP Listener max connections  | `keycloak.ajp.max-connections`     | 500                                                              |
 | IO          | Worker IO thread pool         | `keycloak.worker.io-threads`       | 2                                                                |
@@ -29,35 +29,20 @@
 |-------------|-------------------------------|-------------------------|-----------------------------------------------------------------------------------------|
 | JVM         | Memory settings               | `infinispan.jvm.memory` | -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC |
 
-## CPUs
+## Docker settings
 
-At the moment it is not possible to dynamically parametrize the number of CPUs for a service via Maven properties or environment variables.
+By default, there are 4 CPU cores allocated: core 0 for monitoring, core 1 for database (MariaDB), and cores 2 and 3 for Keycloak server.
+Default memory limits for database and Keycloak server are 2g. The `cpuset` and `memlimit` parameters set here are set to `cpuset` and
+`mem_limit` parameters of docker-compose configuration. See docker-compose documentation for meaning of the values. How to set the parameters
+correctly depends on number of factors - number of cpu cores, NUMA, available memory etc., hence it is out of scope of this document.
 
-To change the default value (`cpus: 1`) it is necessary to edit the Docker Compose file.
+| Container   | Setting                       | Property                        | Default value                                         |
+|-------------|-------------------------------|---------------------------------|-------------------------------------------------------|
+| Keycloak    | Allocated CPUs                | `keycloak.docker.cpuset`        | 2-3                                                   |
+|             | Allocated CPUs for DC1        | `keycloak.dc1.docker.cpuset`    | 2-3                                                   |
+|             | Allocated CPUs for DC2        | `keycloak.dc2.docker.cpuset`    | 2-3                                                   |
+|             | Available memory              | `keycloak.docker.memlimit`      | 2g                                                    |
+| MariaDB     | Allocated CPUs                | `db.docker.cpuset`              | 1                                                     |
+|             | Available memory              | `db.docker.memlimit`            | 2g                                                    |
+| Monitoring  | Allocated CPUs                | `monitoring.docker.cpuset`      | 0                                                     |
 
-
-### Example: Keycloak service using 2 CPU cores
-
-`docker-compose.yml` and `docker-compose-cluster.yml`:
-```
-services:
-    ...
-    keycloak:
-        ...
-        cpus: 2
-        ...
-```
-
-`docker-compose-crossdc.yml`:
-```
-services:
-    ...
-    keycloak_dc1:
-        ...
-        cpus: 2
-        ...
-    keycloak_dc2:
-        ...
-        cpus: 2
-        ...
-```

testsuite/performance/docker-compose-cluster.yml

@@ -15,7 +15,8 @@ services:
     mariadb:
         build: db/mariadb
         image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
+        cpuset: ${DB_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-1g}
         networks:
             - keycloak
         environment:
@@ -32,7 +33,8 @@ services:
         depends_on:
             mariadb:
                 condition: service_healthy
-        cpus: 1
+        cpuset: ${KEYCLOAK_CPUSET:-2-3}
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
         networks:
             - keycloak
         environment:
@@ -46,7 +48,7 @@ services:
             KEYCLOAK_USER: admin
             KEYCLOAK_PASSWORD: admin
 
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
             HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
             AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
             WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}

testsuite/performance/docker-compose-crossdc.yml

@@ -95,7 +95,8 @@ services:
         depends_on: 
             mariadb_dc1:
                 condition: service_healthy
-        cpus: 1
+        cpuset: ${DB_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-1g}
         networks:
             - db_replication
             - dc2_keycloak
@@ -122,7 +123,8 @@ services:
             # wait for the ispn cluster to be ready before starting keycloak
             infinispan_dc2:
                 condition: service_healthy
-        cpus: 1
+        cpuset: ${KEYCLOAK_DC1_CPUSET:-2}
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
         networks:
             - dc1_keycloak
         environment:
@@ -138,7 +140,7 @@ services:
             INFINISPAN_HOST: infinispan_dc1
             SITE: dc1
             
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
             HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
             AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
             WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
@@ -162,7 +164,8 @@ services:
             # wait for first kc instance to be ready before starting another
             keycloak_dc1:
                 condition: service_healthy
-        cpus: 1
+        cpuset: ${KEYCLOAK_DC2_CPUSET:-3}
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
         networks:
             - dc2_keycloak
         environment:
@@ -176,7 +179,7 @@ services:
             INFINISPAN_HOST: infinispan_dc2
             SITE: dc2
 
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
             HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
             AJP_MAX_CONNECTIONS: ${KEYCLOAK_AJP_MAX_CONNECTIONS:-500}
             WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}

testsuite/performance/docker-compose-monitoring.yml

@@ -11,6 +11,7 @@ services:
     
     monitoring_influxdb:
         image: influxdb
+        cpuset: ${MONITORING_CPUSET:-1}
         volumes:
         - influx:/var/lib/influxdb
         networks:
@@ -26,6 +27,7 @@ services:
     monitoring_cadvisor:
         build: monitoring/cadvisor
         image: monitoring_cadvisor
+        cpuset: ${MONITORING_CPUSET:-1}
         hostname: '{{.Node.ID}}'
         volumes:
         - /:/rootfs:ro
@@ -50,6 +52,7 @@ services:
     monitoring_grafana:
         build: monitoring/grafana
         image: monitoring_grafana
+        cpuset: ${MONITORING_CPUSET:-1}
         depends_on:
         - monitoring_influxdb
         volumes:

testsuite/performance/docker-compose.yml

@@ -5,13 +5,14 @@ networks:
         ipam:
             config:
             - subnet: 10.0.1.0/24
-            
+
 services:
-    
+
     mariadb:
         build: db/mariadb
         image: keycloak_test_mariadb:${KEYCLOAK_VERSION:-latest}
-        cpus: 1
+        cpuset: ${DB_CPUSET:-1}
+        mem_limit: ${DB_MEMLIMIT:-1g}
         networks:
             - keycloak
         environment:
@@ -22,14 +23,15 @@ services:
             MYSQL_INITDB_SKIP_TZINFO: 1
         ports:
             - "3306:3306"
-            
+
     keycloak:
         build: keycloak
         image: keycloak_test_keycloak:${KEYCLOAK_VERSION:-latest}
         depends_on:
             mariadb:
                 condition: service_healthy
-        cpus: 1
+        cpuset: ${KEYCLOAK_CPUSET:-2-3}
+        mem_limit: ${KEYCLOAK_MEMLIMIT:-2500m}
         networks:
             - keycloak
         environment:
@@ -40,7 +42,7 @@ services:
             KEYCLOAK_USER: admin
             KEYCLOAK_PASSWORD: admin
             # docker-compose syntax note: ${ENV_VAR:-<DEFAULT_VALUE>}
-            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
+            JAVA_OPTS: ${KEYCLOAK_JVM_MEMORY:--Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
             HTTP_MAX_CONNECTIONS: ${KEYCLOAK_HTTP_MAX_CONNECTIONS:-500}
             WORKER_IO_THREADS: ${KEYCLOAK_WORKER_IO_THREADS:-2}
             WORKER_TASK_MAX_THREADS: ${KEYCLOAK_WORKER_TASK_MAX_THREADS:-16}
@@ -50,4 +52,4 @@ services:
             DS_PS_CACHE_SIZE: ${KEYCLOAK_DS_PS_CACHE_SIZE:-100}
         ports:
             - "8080:8080"
-            - "9990:9990"
+            - "9990:9990"

testsuite/performance/keycloak/Dockerfile

@@ -17,26 +17,23 @@ ADD target/keycloak configs/ ./
 ADD *.sh /usr/local/bin/
 
 USER root
-RUN chown -R jboss .; chgrp -R jboss .; 
+RUN chown -R jboss .; chgrp -R jboss .; \
-RUN chmod -R -v +x /usr/local/bin/
+    chmod -R -v +x /usr/local/bin/ ; \
-RUN yum install -y epel-release jq iproute && yum clean all
+    yum install -y epel-release jq iproute && yum clean all
 
 USER jboss
 # install mariadb JDBC driver
-RUN mkdir -p modules/system/layers/base/org/mariadb/jdbc/main; \
+RUN curl --create-dirs --output modules/system/layers/base/org/mariadb/jdbc/main/mariadb-java-client-2.0.3.jar http://central.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.0.3/mariadb-java-client-2.0.3.jar ; \
-    cd modules/system/layers/base/org/mariadb/jdbc/main; \
+    $JBOSS_HOME/bin/jboss-cli.sh --file=set-keycloak-ds.cli && \
-    curl -O http://central.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.0.3/mariadb-java-client-2.0.3.jar
+    $JBOSS_HOME/bin/jboss-cli.sh --file=io-worker-threads.cli && \
-ADD module.xml modules/system/layers/base/org/mariadb/jdbc/main/
+    $JBOSS_HOME/bin/jboss-cli.sh --file=undertow.cli && \
-# apply configurations
+    $JBOSS_HOME/bin/jboss-cli.sh --file=distributed-cache-owners.cli && \
-RUN $JBOSS_HOME/bin/jboss-cli.sh --file=set-keycloak-ds.cli
+    $JBOSS_HOME/bin/jboss-cli.sh --file=modcluster-simple-load-provider.cli && \
-RUN $JBOSS_HOME/bin/jboss-cli.sh --file=io-worker-threads.cli
+    if [ "$REMOTE_CACHES" == "true" ]; then $JBOSS_HOME/bin/jboss-cli.sh --file=add-remote-cache-stores.cli; fi && \
-RUN $JBOSS_HOME/bin/jboss-cli.sh --file=undertow.cli
+    cd $JBOSS_HOME/standalone; rm -rf configuration/standalone_xml_history log data tmp ; \
-RUN $JBOSS_HOME/bin/jboss-cli.sh --file=distributed-cache-owners.cli
+    $JBOSS_HOME/bin/add-user.sh -u $DEBUG_USER -p $DEBUG_USER_PASSWORD
-RUN $JBOSS_HOME/bin/jboss-cli.sh --file=modcluster-simple-load-provider.cli
-RUN if [ "$REMOTE_CACHES" == "true" ]; then $JBOSS_HOME/bin/jboss-cli.sh --file=add-remote-cache-stores.cli; fi
-RUN cd $JBOSS_HOME/standalone; rm -rf configuration/standalone_xml_history log data tmp
 
-RUN $JBOSS_HOME/bin/add-user.sh -u $DEBUG_USER -p $DEBUG_USER_PASSWORD
+ADD module.xml modules/system/layers/base/org/mariadb/jdbc/main/
 
 EXPOSE 8080
 EXPOSE 9990

testsuite/performance/keycloak/docker-entrypoint.sh

@@ -15,4 +15,3 @@ if [ $KEYCLOAK_USER ] && [ $KEYCLOAK_PASSWORD ]; then
 fi
 
 exec /opt/jboss/keycloak/bin/standalone.sh $PARAMS
-exit $?

testsuite/performance/prepare-dump.sh

@@ -4,13 +4,7 @@ GATLING_HOME=$DIRNAME/tests
 
 if [ -z "$DATASET" ]; then
     echo "This script requires DATASET env variable to be set"
-    echo 1
+    exit 1
-fi
-
-./prepare-data.sh $@
-if [ $? -ne 0 ]; then
-    echo "Failed! See log file for details."
-    exit $?
 fi
 
 echo "Exporting dump file"

testsuite/performance/tests/pom.xml

@@ -39,7 +39,7 @@
         <keycloak.server.uris>http://localhost:8080/auth</keycloak.server.uris>
         <db.url>jdbc:mariadb://keycloak:keycloak@localhost:3306/keycloak</db.url>
 
-        <keycloak.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</keycloak.jvm.memory>
+        <keycloak.jvm.memory>-Xms64m -Xmx2g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</keycloak.jvm.memory>
         <keycloak.http.max-connections>500</keycloak.http.max-connections>
         <keycloak.ajp.max-connections>500</keycloak.ajp.max-connections>
         <keycloak.worker.io-threads>2</keycloak.worker.io-threads>
@@ -48,14 +48,25 @@
         <keycloak.ds.max-pool-size>100</keycloak.ds.max-pool-size>
         <keycloak.ds.pool-prefill>true</keycloak.ds.pool-prefill>
         <keycloak.ds.ps-cache-size>100</keycloak.ds.ps-cache-size>
-        
+
         <keycloak-lb.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m</keycloak-lb.jvm.memory>
         <keycloak-lb.http.max-connections>500</keycloak-lb.http.max-connections>
         <keycloak-lb.worker.io-threads>2</keycloak-lb.worker.io-threads>
         <keycloak-lb.worker.task-max-threads>16</keycloak-lb.worker.task-max-threads>
-        
+
+        <!-- Docker-related properties -->
+        <db.docker.cpuset>1</db.docker.cpuset>
+        <keycloak.docker.cpuset>2-3</keycloak.docker.cpuset>
+        <keycloak.dc1.docker.cpuset>2</keycloak.dc1.docker.cpuset>
+        <keycloak.dc2.docker.cpuset>3</keycloak.dc2.docker.cpuset>
+        <monitoring.docker.cpuset>0</monitoring.docker.cpuset>
+
+        <db.docker.memlimit>2g</db.docker.memlimit>
+        <keycloak.docker.memlimit>2g</keycloak.docker.memlimit>
+        <!-- End of docker-related properties -->
+
         <infinispan.jvm.memory>-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC</infinispan.jvm.memory>
-        
+
         <dataset>default</dataset>
         <numOfWorkers>1</numOfWorkers>
 
@@ -321,6 +332,13 @@
                                     <commandlineArgs>-f ${compose.file} up -d --build ${compose.up.params}</commandlineArgs>
                                     <environmentVariables>
                                         <KEYCLOAK_VERSION>${project.version}</KEYCLOAK_VERSION>
+
+                                        <KEYCLOAK_CPUSET>${keycloak.docker.cpuset}</KEYCLOAK_CPUSET>
+                                        <KEYCLOAK_DC1_CPUSET>${keycloak.dc1.docker.cpuset}</KEYCLOAK_DC1_CPUSET>
+                                        <KEYCLOAK_DC2_CPUSET>${keycloak.dc2.docker.cpuset}</KEYCLOAK_DC2_CPUSET>
+                                        <KEYCLOAK_MEMLIMIT>${keycloak.docker.memlimit}</KEYCLOAK_MEMLIMIT>
+                                        <DB_CPUSET>${db.docker.cpuset}</DB_CPUSET>
+                                        <DB_MEMLIMIT>${db.docker.memlimit}</DB_MEMLIMIT>
                                         
                                         <KEYCLOAK_JVM_MEMORY>${keycloak.jvm.memory}</KEYCLOAK_JVM_MEMORY>
                                         <KEYCLOAK_HTTP_MAX_CONNECTIONS>${keycloak.http.max-connections}</KEYCLOAK_HTTP_MAX_CONNECTIONS>
@@ -462,6 +480,43 @@
             </build>
         </profile>
 
+        <profile>
+            <id>export-dump-after-generation</id>
+
+            <activation>
+                <activeByDefault>false</activeByDefault>
+                <property>
+                    <name>export-dump</name>
+                </property>
+            </activation>
+
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.codehaus.mojo</groupId>
+                        <artifactId>exec-maven-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>export-dump</id>
+                                <phase>pre-integration-test</phase>
+                                <goals>
+                                    <goal>exec</goal>
+                                </goals>
+                                <configuration>
+                                    <workingDirectory>${project.basedir}/..</workingDirectory>
+                                    <executable>./prepare-dump.sh</executable>
+
+                                    <environmentVariables>
+                                        <DATASET>${dataset}</DATASET>
+                                    </environmentVariables>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
+
         <profile>
             <id>import-dump</id>
             <build>
@@ -581,6 +636,9 @@
                                     <workingDirectory>${project.basedir}/..</workingDirectory>
                                     <executable>docker-compose</executable>
                                     <commandlineArgs>-f docker-compose-monitoring.yml up -d --build</commandlineArgs>
+                                    <environmentVariables>
+                                        <MONITORING_CPUSET>${monitoring.docker.cpuset}</MONITORING_CPUSET>
+                                    </environmentVariables>
                                 </configuration>
                             </execution>
                         </executions>