KEYCLOAK-5955 Add ability to remotely monitor Infinispan and load balancer

testsuite/performance/README.md

@@ -135,26 +135,30 @@ For example:
 `mvn verify -Ptest -DrunUsers=1 -DnumOfIterations=10 -DuserThinkTime=0 -Ddataset=100u -DrefreshTokenPeriod=10 -Dgatling.simulationClass=keycloak.AdminSimulation`
 
 
-## Debugging & Profiling
+## Monitoring
 
-Keycloak docker container exposes JMX management interface on port `9990`.
+### JMX
 
-### JVisualVM
+To enable access to JMX on the WildFly-backed services set properties `management.user` and `management.user.password` during the provisioning phase.
 
+#### JVisualVM
+
+- Set `JBOSS_HOME` variable to point to a valid WildFly 10+ installation.
 - Start JVisualVM with `jboss-client.jar` on classpath: `./jvisualvm --cp:a $JBOSS_HOME/bin/client/jboss-client.jar`.
-- Add a local JMX connection: `service:jmx:remote+http://localhost:9990`.
+- Add a local JMX connection: `service:jmx:remote+http://localhost:9990`. <sup>**[*]**</sup>
 - Check "Use security credentials" and set `admin:admin`. (The default credentials can be overriden by providing env. variables `DEBUG_USER` and `DEBUG_USER_PASSWORD` to the container.)
 - Open the added connection.
 
-_Note: The above applies for the singlenode deployment.
+**[*]** For `singlenode` this points to the JMX console of the Keycloak server.
-In cluster/crossdc deployments there are multiple KC containers running at the same time so their exposed ports are mapped to random available ports on `0.0.0.0`.
+To get the connection URLs for `cluster` or `crossdc` deployments see the JMX section in the generated `provisioned-system.properties` file.
-To find the actual mapped ports run command: `docker ps | grep performance_keycloak`._
+- Property `keycloak.frontend.servers.jmx` contains JMX URLs of the Load Balancers.
+- Property `keycloak.backend.servers.jmx` contains JMX URLs of the clustered Keycloak servers.
+- Property `infinispan.servers.jmx` contains JMX URLs of the Infinispan servers, in Cross-DC deployment.
 
+### Docker Monitoring
 
-## Monitoring
+There is a docker-based solution for monitoring CPU, memory and network usage per container. 
-
+It uses CAdvisor service to export container metrics into InfluxDB time series database, and Grafana web app to query the DB and present results as graphs.
-There is a docker-based solution for monitoring of CPU, memory and network usage per container. 
-(It uses CAdvisor service to export container metrics into InfluxDB time series database, and Grafana web app to query the DB and present results as graphs.)
 
 - To enable run: `mvn verify -Pmonitoring`
 - To disable run: `mvn verify -Pmonitoring-off[,delete-monitoring-data]`.

testsuite/performance/README.provisioning-parameters.md

@@ -82,6 +82,13 @@ The maximum cluster size corresponds to the number of cpusets.
 | Category    | Setting                       | Property                    | Default Value   |
 |-------------|-------------------------------|-----------------------------|-----------------|
 | Docker      | Allocated CPUs                | `monitoring.docker.cpusets` | `0`             |
+| JMX         | Management user               | `management.user`           | Not set.        |
+|             | Management user's password    | `management.user.password`  | Not set.        |
+
+By setting the `managemen.user` and `management.user.password` parameters it is possible 
+to add a management user to all WildFly-backed services (*Keycloak Server*, *Infinispan Server* and the *Load Balancer*).
+Unless both parameters are explicitly provided during the provisioning phase the user will not be added 
+and it won't be possible to log into the management console or access JMX.
 
 
 ## Note on Docker settings

testsuite/performance/infinispan/Dockerfile

@@ -4,6 +4,9 @@ FROM jboss/infinispan-server:8.2.6.Final
 ARG LOCAL_SITE
 ARG REMOTE_SITE
 
+ARG MANAGEMENT_USER
+ARG MANAGEMENT_USER_PASS
+
 USER root
 RUN yum -y install iproute
 USER jboss
@@ -19,6 +22,7 @@ USER root
 RUN chmod -v +x /usr/local/bin/*.sh
 USER jboss
 
+RUN if [ ! -z "$MANAGEMENT_USER" ]; then $INFINISPAN_SERVER_HOME/bin/add-user.sh -u $MANAGEMENT_USER -p $MANAGEMENT_USER_PASS ; fi
 RUN $INFINISPAN_SERVER_HOME/bin/ispn-cli.sh --file=add-private-network-interface.cli; \
     $INFINISPAN_SERVER_HOME/bin/ispn-cli.sh --file=add-keycloak-caches.cli; \
     cd $INFINISPAN_SERVER_HOME/standalone; rm -rf configuration/standalone_xml_history log data tmp

testsuite/performance/keycloak/configure.xml

@@ -1,12 +1,16 @@
 <project name="keycloak-server-configuration" basedir="." >
 
-    <target name="check-if-performance-configured">
+    <target name="check-configuration-state">
         <available property="performance.configured" file="${project.build.directory}/performance-configured"/>
+        <available property="management.configured" file="${project.build.directory}/management-configured"/>
+        <available property="crossdc.configured" file="${project.build.directory}/crossdc-configured"/>
         <echo>performance.configured: ${performance.configured}</echo>
+        <echo>management.configured: ${management.configured}</echo>
+        <echo>crossdc.configured: ${crossdc.configured}</echo>
     </target>                            
     
-    <target name="keycloak-performance-configuration" unless="performance.configured" depends="check-if-performance-configured">
+    <target name="keycloak-performance-configuration" unless="performance.configured" depends="check-configuration-state">
-        <echo>keycloak-performance-configuration</echo>
+        <echo>Applying keycloak performance configuration.</echo>
         <chmod perm="ug+x">
             <fileset dir="${server.unpacked.home}/bin">
                 <include name="*.sh"/>
@@ -23,41 +27,53 @@
         </copy>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=set-keycloak-ds.cli"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
         </exec>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=io-worker-threads.cli"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
         </exec>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=undertow.cli"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
         </exec>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=modcluster-simple-load-provider.cli"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
         </exec>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=io-worker-threads.cli"/>
-        </exec>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
-        <exec executable="./${add.user.script}" dir="${server.unpacked.home}/bin" failonerror="true">
-            <arg value="-u"/>
-            <arg value="${keycloak.debug.user}"/>
-            <arg value="-p"/>
-            <arg value="${keycloak.debug.user.password}"/>
         </exec>
         <delete dir="${server.unpacked.home}/standalone/configuration/standalone_xml_history"/>
+        <delete dir="${server.unpacked.home}/standalone/log"/>
+        <delete dir="${server.unpacked.home}/standalone/data"/>
+        <delete dir="${server.unpacked.home}/standalone/tmp"/>
         <touch file="${project.build.directory}/performance-configured"/>
     </target>
 
-        
+    <target name="add-management-user" unless="management.configured" depends="check-configuration-state">
-    <target name="check-if-crossdc-configured">
+        <echo>Adding management user: `${management.user}`</echo>
-        <available property="crossdc.configured" file="${project.build.directory}/crossdc-configured"/>
+        <exec executable="./${add.user.script}" dir="${server.unpacked.home}/bin" failonerror="true">
-        <echo>crossdc.configured: ${crossdc.configured}</echo>
+            <arg value="-u"/>
+            <arg value="${management.user}"/>
+            <arg value="-p"/>
+            <arg value="${management.user.password}"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
+        </exec>
+        <touch file="${project.build.directory}/management-configured"/>
     </target>
     
-    <target name="keycloak-crossdc-configuration" unless="crossdc.configured" depends="check-if-crossdc-configured">
+    <target name="keycloak-crossdc-configuration" unless="crossdc.configured" depends="check-configuration-state">
         <echo>keycloak-crossdc-configuration</echo>
         <exec executable="./${jboss.cli.script}" dir="${server.unpacked.home}/bin" failonerror="true">
             <arg value="--file=add-remote-cache-stores.cli"/>
+            <env key="JBOSS_HOME" value="${server.unpacked.home}"/>
         </exec>
         <delete dir="${server.unpacked.home}/standalone/configuration/standalone_xml_history"/>
+        <delete dir="${server.unpacked.home}/standalone/log"/>
+        <delete dir="${server.unpacked.home}/standalone/data"/>
+        <delete dir="${server.unpacked.home}/standalone/tmp"/>
         <touch file="${project.build.directory}/crossdc-configured"/>
     </target>
 

testsuite/performance/keycloak/pom.xml

@@ -51,11 +51,9 @@
         <skip.crossdc.configuration>true</skip.crossdc.configuration>
         
         <skip.configuration>false</skip.configuration>
+        <skip.add.management.user>true</skip.add.management.user>
         <skip.keycloak.docker>false</skip.keycloak.docker>
         
-        <keycloak.debug.user>admin</keycloak.debug.user>
-        <keycloak.debug.user.password>admin</keycloak.debug.user.password>
-        
         <scripts.dir>${project.build.scriptSourceDirectory}</scripts.dir>
         <resources.dir>${project.basedir}/src/main/resources</resources.dir>
     </properties>
@@ -135,6 +133,19 @@
                             </target>
                         </configuration>
                     </execution>
+                    <execution>
+                        <id>add-management-user</id>
+                        <phase>process-resources</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <skip>${skip.add.management.user}</skip>
+                            <target>
+                                <ant antfile="configure.xml" target="add-management-user" />
+                            </target>
+                        </configuration>
+                    </execution>
                     <execution>
                         <id>keycloak-docker</id>
                         <phase>process-resources</phase>
@@ -167,6 +178,22 @@
             </properties>
         </profile>
         
+        <profile>        
+            <id>add-management-user</id>
+            <activation>
+                <property>
+                    <name>management.user</name>
+                </property>
+            </activation>
+            <properties>
+                <skip.add.management.user>false</skip.add.management.user>
+                <!--it seems to be necessary to explicitly re-set these properties here
+                otherwise the antrun plugin won't pick them up-->
+                <management.user>${management.user}</management.user>
+                <management.user.password>${management.user.password}</management.user.password>
+            </properties>
+        </profile>
+        
         <profile>
             <id>crossdc</id>
             <properties>

testsuite/performance/pom.xml

@@ -31,6 +31,11 @@
     <name>Keycloak Performance TestSuite</name>
     <packaging>pom</packaging>
     
+    <properties>
+        <management.user/>
+        <management.user.password/>
+    </properties>
+            
     <modules>
         <module>keycloak</module>
         <module>tests</module>

testsuite/performance/tests/docker-compose.sh

@@ -52,13 +52,20 @@ function inspectDockerPortMapping() {
 function generateProvisionedSystemProperties() {
     echo "Generating $PROVISIONED_SYSTEM_PROPERTIES_FILE"
     echo "deployment=$DEPLOYMENT" > $PROVISIONED_SYSTEM_PROPERTIES_FILE
+    echo "# Docker Compose" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
     echo "keycloak.docker.services=$KEYCLOAK_SERVICES" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
     case "$DEPLOYMENT" in
         singlenode)
+            echo "# HTTP" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
             inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_keycloak_1
             echo "keycloak.frontend.servers=http://localhost:$MAPPED_PORT/auth" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+
+            echo "# JMX" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_keycloak_1
+            echo "keycloak.frontend.servers.jmx=service:jmx:remote+http://localhost:$MAPPED_PORT" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
         ;;
         cluster)
+            echo "# HTTP" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
             inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_1
             echo "keycloak.frontend.servers=http://localhost:$MAPPED_PORT/auth" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
             BACKEND_URLS=""
@@ -67,8 +74,19 @@ function generateProvisionedSystemProperties() {
                 BACKEND_URLS="$BACKEND_URLS http://localhost:$MAPPED_PORT/auth"
             done
             echo "keycloak.backend.servers=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+
+            echo "# JMX" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_loadbalancer_1
+            echo "keycloak.frontend.servers.jmx=service:jmx:remote+http://localhost:$MAPPED_PORT" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            BACKEND_URLS=""
+            for SERVICE in $KEYCLOAK_SERVICES ; do
+                inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_${SERVICE}_1
+                BACKEND_URLS="$BACKEND_URLS service:jmx:remote+http://localhost:$MAPPED_PORT"
+            done
+            echo "keycloak.backend.servers.jmx=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
         ;;
         crossdc) 
+            echo "# HTTP" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
             inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_dc1_1
             KC_DC1_PORT=$MAPPED_PORT
             inspectDockerPortMapping 8080/tcp ${PROJECT_NAME}_loadbalancer_dc2_1
@@ -80,6 +98,25 @@ function generateProvisionedSystemProperties() {
                 BACKEND_URLS="$BACKEND_URLS http://localhost:$MAPPED_PORT/auth"
             done
             echo "keycloak.backend.servers=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+
+            echo "# JMX" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_loadbalancer_dc1_1
+            KC_DC1_PORT=$MAPPED_PORT
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_loadbalancer_dc2_1
+            KC_DC2_PORT=$MAPPED_PORT
+            echo "keycloak.frontend.servers.jmx=service:jmx:remote+http://localhost:$KC_DC1_PORT service:jmx:remote+http://localhost:$KC_DC2_PORT" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+            BACKEND_URLS=""
+            for SERVICE in $KEYCLOAK_SERVICES ; do
+                inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_${SERVICE}_1
+                BACKEND_URLS="$BACKEND_URLS service:jmx:remote+http://localhost:$MAPPED_PORT"
+            done
+            echo "keycloak.backend.servers.jmx=$BACKEND_URLS" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
+
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_infinispan_dc1_1
+            ISPN_DC1_PORT=$MAPPED_PORT
+            inspectDockerPortMapping 9990/tcp ${PROJECT_NAME}_infinispan_dc2_1
+            ISPN_DC2_PORT=$MAPPED_PORT
+            echo "infinispan.servers.jmx=service:jmx:remote+http://localhost:$ISPN_DC1_PORT service:jmx:remote+http://localhost:$ISPN_DC2_PORT" >> $PROVISIONED_SYSTEM_PROPERTIES_FILE
         ;;
     esac
 }

testsuite/performance/tests/pom.xml

@@ -391,6 +391,9 @@
 
                                         <KEYCLOAK_VERSION>${project.version}</KEYCLOAK_VERSION>
                                         
+                                        <MANAGEMENT_USER>${management.user}</MANAGEMENT_USER>
+                                        <MANAGEMENT_USER_PASS>${management.user.password}</MANAGEMENT_USER_PASS>
+                                        
                                         <KEYCLOAK_SCALE>${keycloak.scale}</KEYCLOAK_SCALE>
                                         <KEYCLOAK_DC1_SCALE>${keycloak.dc1.scale}</KEYCLOAK_DC1_SCALE>
                                         <KEYCLOAK_DC2_SCALE>${keycloak.dc2.scale}</KEYCLOAK_DC2_SCALE>

testsuite/performance/tests/src/main/docker-compose/cluster/docker-compose-base.yml

@@ -47,5 +47,6 @@ services:
             WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
         ports:
             - "8080:8080"
+            - "9990:9990"
 
 

testsuite/performance/tests/src/main/docker-compose/crossdc/docker-compose-base.yml

@@ -35,6 +35,8 @@ services:
             args:
                 LOCAL_SITE: dc1
                 REMOTE_SITE: dc2
+                MANAGEMENT_USER: ${MANAGEMENT_USER}
+                MANAGEMENT_USER_PASS: ${MANAGEMENT_USER_PASS}
         image: keycloak_test_infinispan_dc1:${KEYCLOAK_VERSION:-latest}
         cpuset: ${INFINISPAN_DC1_CPUSET:-1}
         mem_limit: ${INFINISPAN_MEMLIMIT:-1500m}
@@ -49,7 +51,7 @@ services:
             TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600]
             JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
         ports:
-            - "9991:9990"
+            - "9990"
 
     infinispan_dc2:
         build: 
@@ -57,6 +59,8 @@ services:
             args:
                 LOCAL_SITE: dc2
                 REMOTE_SITE: dc1
+                MANAGEMENT_USER: ${MANAGEMENT_USER}
+                MANAGEMENT_USER_PASS: ${MANAGEMENT_USER_PASS}
         image: keycloak_test_infinispan_dc2:${KEYCLOAK_VERSION:-latest}
         depends_on:
             infinispan_dc1:
@@ -74,7 +78,7 @@ services:
             TCP_PING_INITIAL_HOSTS: infinispan_dc1[7600],infinispan_dc2[7600]
             JAVA_OPTS: ${INFINISPAN_JVM_MEMORY:--Xms64m -Xmx1g -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC} -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
         ports:
-            - "9992:9990"
+            - "9990"
     
     
     mariadb_dc1:
@@ -94,7 +98,7 @@ services:
         entrypoint: docker-entrypoint-wsrep.sh
         command: --wsrep-new-cluster
         ports:
-            - "3306:3306"
+            - "3307:3306"
             
     mariadb_dc2:
         build: db/mariadb
@@ -113,7 +117,7 @@ services:
         entrypoint: docker-entrypoint-wsrep.sh
         command: --wsrep_cluster_address=gcomm://mariadb_dc1
         ports:
-            - "3307:3306"
+            - "3308:3306"
 
 
     loadbalancer_dc1:
@@ -133,6 +137,7 @@ services:
             WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
         ports:
             - "8081:8080"
+            - "9991:9990"
             
     loadbalancer_dc2:
         build: load-balancer/wildfly-modcluster
@@ -151,4 +156,5 @@ services:
             WORKER_TASK_MAX_THREADS: ${LB_WORKER_TASK_MAX_THREADS:-16}
         ports:
             - "8082:8080"
+            - "9992:9990"