KEYCLOAK-9456 Docker support for testing with MSSQL, Oracle 11g

misc/DatabaseTesting.md

@@ -61,14 +61,36 @@ The project provides specific profiles to run database tests using containers. T
 * `db-mysql`
 * `db-postgres`
 * `db-mariadb`
+* `db-mssql2017`
+* `db-oracle11g`
 
-As an example, to run tests using a MySQL docker container:
+As an example, to run tests using a MySQL docker container on Undertow auth-server:
 
-    mvn -f testsuite/integration-arquillian clean verify -Pdb-mysql,jpa
+    mvn -f testsuite/integration-arquillian clean verify -Pdb-mysql
 
 If you want to run tests using a pre-configured Keycloak distribution (instead of Undertow):
 
     mvn -f testsuite/integration-arquillian clean verify -Pdb-mysql,jpa,auth-server-wildfly
-    
+
-Note that you must always activate the `jpa` profile.
+Note that you must always activate the `jpa` profile when using auth-server-wildfly.
-    
+
+If the mvn command fails for any reason, it may also fail to remove the container which
+must be then removed manually.
+
+For Oracle databases, neither JDBC driver nor the image are publicly available
+due to licensing restrictions and require preparation of the environment. You
+first need to download the JDBC driver and install it to your local maven repo
+(feel free to specify GAV and file according to the one you would download):
+
+    mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc7 -Dversion=12.1.0 -Dpackaging=jar -Dfile=ojdbc7.jar -DgeneratePom=true
+
+Then build the Docker image per instructions at
+https://github.com/oracle/docker-images/tree/master/OracleDatabase. The last
+step is running which might require updating the `jdbc.mvn.groupId`,
+`jdbc.mvn.artifactId`, and `jdbc.mvn.version` according to the parameters you
+used in the command above, and `docker.database.image` if you used a different
+name or tag for the image.
+
+Note that Docker containers may occupy some space even after termination, and
+especially with databases that might be easily a gigabyte. It is thus
+advisable to run `docker system prune` occasionally to reclaim that space.

pom.xml

@@ -114,6 +114,7 @@
         <pax.web.version>7.1.0</pax.web.version>
         <postgresql.version>9.3-1100-jdbc41</postgresql.version>
         <mariadb.version>2.2.4</mariadb.version>
+        <mssql.version>7.0.0.jre8</mssql.version>
         <servlet.api.30.version>1.0.2.Final</servlet.api.30.version>
         <servlet.api.40.version>1.0.0.Final</servlet.api.40.version>
         <twitter4j.version>4.0.4</twitter4j.version>

testsuite/integration-arquillian/HOW-TO-RUN.md

@@ -519,24 +519,10 @@ The setup includes:
 
 ### Cluster tests with Keycloak on Wildfly
 
-After build the sources, distribution and setup of clean shared database (replace command according your DB), you can use this command to setup servers:
+After you build the distribution, you run this command to setup servers and run cluster tests using shared Docker database:
 
-    export DB_HOST=localhost
+    mvn -f testsuite/integration-arquillian/pom.xml \
-    mvn -f testsuite/integration-arquillian/servers/pom.xml \
+    -Pauth-server-wildfly,auth-server-cluster,db-mysql,jpa \
-    -Pauth-server-wildfly,auth-server-cluster,jpa \
-    -Dsession.cache.owners=2 \
-    -Djdbc.mvn.groupId=mysql \
-    -Djdbc.mvn.version=5.1.29 \
-    -Djdbc.mvn.artifactId=mysql-connector-java \
-    -Dkeycloak.connectionsJpa.url=jdbc:mysql://$DB_HOST/keycloak \
-    -Dkeycloak.connectionsJpa.user=keycloak \
-    -Dkeycloak.connectionsJpa.password=keycloak \
-    clean install
-    
-And then this to run the cluster tests:
-   
-    mvn -f testsuite/integration-arquillian/tests/base/pom.xml \
-    -Pauth-server-wildfly,auth-server-cluster \
     -Dsession.cache.owners=2 \
     -Dbackends.console.output=true \
     -Dauth.server.log.check=false \
@@ -547,15 +533,12 @@ And then this to run the cluster tests:
 ### Cluster tests with Keycloak on embedded undertow
 
     mvn -f testsuite/integration-arquillian/tests/base/pom.xml \
-    -Pauth-server-cluster-undertow \
+    -Pauth-server-cluster-undertow,db-mysql \
     -Dsession.cache.owners=2 \
     -Dkeycloak.connectionsInfinispan.sessionsOwners=2 \
     -Dbackends.console.output=true \
     -Dauth.server.log.check=false \
     -Dfrontend.console.output=true \
-    -Dkeycloak.connectionsJpa.url=jdbc:mysql://$DB_HOST/keycloak \
-    -Dkeycloak.connectionsJpa.user=keycloak \
-    -Dkeycloak.connectionsJpa.password=keycloak \
     -Dtest=org.keycloak.testsuite.cluster.**.*Test clean install
 
 #### Run cluster tests from IDE on embedded undertow
@@ -619,9 +602,9 @@ a2) If you want to use **JBoss-based** Keycloak backend containers instead of co
 
 *note: 'auth-server-wildfly' can be replaced by 'auth-server-eap'*
 
-By default JBoss-based containers use TCP-based h2 database. It can be configured to use real DB, e.g. with following command:
+By default JBoss-based containers use TCP-based h2 database. It can be configured to use real DB spawn in Docker, e.g. with following command:
 
-  `mvn -Pcache-server-infinispan,auth-servers-crossdc-jboss,auth-server-wildfly,jpa -f testsuite/integration-arquillian -DskipTests clean install -Djdbc.mvn.groupId=org.mariadb.jdbc -Djdbc.mvn.artifactId=mariadb-java-client -Djdbc.mvn.version=2.0.3 -Dkeycloak.connectionsJpa.url=jdbc:mariadb://localhost:3306/keycloak -Dkeycloak.connectionsJpa.password=keycloak -Dkeycloak.connectionsJpa.user=keycloak`
+  `mvn -Pcache-server-infinispan,auth-servers-crossdc-jboss,auth-server-wildfly,jpa,db-mariadb -f testsuite/integration-arquillian -DskipTests clean install`
 
 b1) For **Undertow** Keycloak backend containers, you can run the tests using the following command (adjust the test specification according to your needs):
 
@@ -644,7 +627,7 @@ b2) For **JBoss-based** Keycloak backend containers, you can run the tests like
 **note**:
 For **JBoss-based** Keycloak backend containers on real DB, the previous commands from (a2) and (b2) can be "squashed" into one. E.g.:
 
-  `mvn -f testsuite/integration-arquillian clean install -Dtest=*.crossdc.* -Djdbc.mvn.groupId=org.mariadb.jdbc -Djdbc.mvn.artifactId=mariadb-java-client -Djdbc.mvn.version=2.0.3 -Dkeycloak.connectionsJpa.url=jdbc:mariadb://localhost:3306/keycloak -Dkeycloak.connectionsJpa.password=keycloak -Dkeycloak.connectionsJpa.user=keycloak -Pcache-server-infinispan,auth-servers-crossdc-jboss,auth-server-wildfly,jpa clean install`
+  `mvn -f testsuite/integration-arquillian -Dtest=*.crossdc.* -Pcache-server-infinispan,auth-servers-crossdc-jboss,auth-server-wildfly,jpa,db-mariadb clean install`
     
 
 #### Run Cross-DC Tests from Intellij IDEA

testsuite/integration-arquillian/pom.xml

@@ -75,6 +75,19 @@
 
         <!-- By default, skip docker-maven-plugin when running base tests-->
         <docker.database.skip>true</docker.database.skip>
+        <docker.database.postStart>/bin/true</docker.database.postStart>
+        <docker.database.wait-for-log-regex>NEVER-MATCHING-REGEX</docker.database.wait-for-log-regex>
+        <docker.database.shmsize>67108864</docker.database.shmsize>
+
+        <jdbc.mvn.groupId>com.h2database</jdbc.mvn.groupId>
+        <jdbc.mvn.artifactId>h2</jdbc.mvn.artifactId>
+        <jdbc.mvn.version>${h2.version}</jdbc.mvn.version>
+
+        <keycloak.connectionsJpa.driver>org.h2.Driver</keycloak.connectionsJpa.driver>
+        <keycloak.connectionsJpa.database>keycloak</keycloak.connectionsJpa.database>
+        <keycloak.connectionsJpa.user>sa</keycloak.connectionsJpa.user>
+        <keycloak.connectionsJpa.password></keycloak.connectionsJpa.password>
+        <keycloak.connectionsJpa.url>jdbc:h2:mem:test;MVCC=TRUE;DB_CLOSE_DELAY=-1</keycloak.connectionsJpa.url>
     </properties>
 
     <dependencyManagement>
@@ -337,6 +350,7 @@
                 <docker.database.image>mysql:5.7.25</docker.database.image>
                 <docker.database.port>3306</docker.database.port>
                 <docker.database.skip>false</docker.database.skip>
+                <docker.database.wait-for-log-regex>(?si)Ready for start up.*ready [^\n]{0,30}connections</docker.database.wait-for-log-regex>
             </properties>
         </profile>
         <profile>
@@ -353,6 +367,7 @@
                 <docker.database.image>postgres:9.6.11</docker.database.image>
                 <docker.database.port>5432</docker.database.port>
                 <docker.database.skip>false</docker.database.skip>
+                <docker.database.wait-for-log-regex>(?si)Ready for start up.*ready [^\n]{0,30}connections</docker.database.wait-for-log-regex>
             </properties>
         </profile>
         <profile>
@@ -369,6 +384,44 @@
                 <docker.database.image>mariadb:10.2.21</docker.database.image>
                 <docker.database.port>3306</docker.database.port>
                 <docker.database.skip>false</docker.database.skip>
+                <docker.database.wait-for-log-regex>(?si)Ready for start up.*ready [^\n]{0,30}connections</docker.database.wait-for-log-regex>
+            </properties>
+        </profile>
+        <profile>
+            <id>db-mssql2017</id>
+            <properties>
+                <docker.database.image>microsoft/mssql-server-linux:2017-GA</docker.database.image>
+                <docker.database.port>1433</docker.database.port>
+                <docker.database.skip>false</docker.database.skip>
+                <docker.database.postStart>/opt/mssql-tools/bin/sqlcmd -e -U sa -P vEry5tron9Pwd -d master -Q CREATE\ DATABASE\ ${keycloak.connectionsJpa.database}</docker.database.postStart>
+                <docker.database.wait-for-log-regex>(?si)SQL Server is now ready for client connections.*Service Broker manager has started</docker.database.wait-for-log-regex>
+                <keycloak.connectionsJpa.driver>com.microsoft.sqlserver.jdbc.SQLServerDriver</keycloak.connectionsJpa.driver>
+                <keycloak.connectionsJpa.database>keycloak</keycloak.connectionsJpa.database>
+                <keycloak.connectionsJpa.user>sa</keycloak.connectionsJpa.user>
+                <keycloak.connectionsJpa.password>vEry5tron9Pwd</keycloak.connectionsJpa.password>
+                <keycloak.connectionsJpa.url>jdbc:sqlserver://${auth.server.db.host}:${docker.database.port};databaseName=${keycloak.connectionsJpa.database}</keycloak.connectionsJpa.url>
+                <jdbc.mvn.groupId>com.microsoft.sqlserver</jdbc.mvn.groupId>
+                <jdbc.mvn.artifactId>mssql-jdbc</jdbc.mvn.artifactId>
+                <jdbc.mvn.version>${mssql.version}</jdbc.mvn.version>
+            </properties>
+        </profile>
+        <profile>
+            <id>db-oracle11g</id>
+            <properties>
+                <docker.database.image>oracle/database:11.2.0.2-xe</docker.database.image>
+                <docker.database.port>1521</docker.database.port>
+                <docker.database.shmsize>1073741824</docker.database.shmsize>
+                <docker.database.skip>false</docker.database.skip>
+                <docker.database.wait-for-log-regex>(?si)DATABASE IS READY TO USE</docker.database.wait-for-log-regex>
+                <keycloak.connectionsJpa.driver>oracle.jdbc.OracleDriver</keycloak.connectionsJpa.driver>
+                <keycloak.connectionsJpa.database>XE</keycloak.connectionsJpa.database>
+                <keycloak.connectionsJpa.user>keycloak</keycloak.connectionsJpa.user>
+                <keycloak.connectionsJpa.password>keycloak</keycloak.connectionsJpa.password>
+                <keycloak.connectionsJpa.url>jdbc:oracle:thin:@${auth.server.db.host}:${docker.database.port}:${keycloak.connectionsJpa.database}</keycloak.connectionsJpa.url>
+                <docker.database.postStart>bash -c while\ !\ sqlplus\ -L\ SYS/sa@localhost/XE\ AS\ SYSDBA\ &lt;&lt;&lt;\ $'CREATE\ USER\ ${keycloak.connectionsJpa.user}\ IDENTIFIED\ BY\ ${keycloak.connectionsJpa.password};\n\ GRANT\ CONNECT,\ RESOURCE,\ DBA,\ GRANT\ ANY\ PRIVILEGE,\ UNLIMITED\ TABLESPACE\ TO\ ${keycloak.connectionsJpa.user};\n';\ do\ sleep\ 5;\ done</docker.database.postStart>
+                <jdbc.mvn.groupId>com.oracle</jdbc.mvn.groupId>
+                <jdbc.mvn.artifactId>ojdbc7</jdbc.mvn.artifactId>
+                <jdbc.mvn.version>12.1.0</jdbc.mvn.version>
             </properties>
         </profile>
     </profiles>

testsuite/integration-arquillian/tests/base/pom.xml

@@ -350,6 +350,7 @@
                                     <alias>testdb</alias>
                                     <name>${docker.database.image}</name>
                                     <run>
+                                        <shmSize>${docker.database.shmsize}</shmSize>
                                         <ports>
                                             <port>${docker.database.port}</port>
                                         </ports>
@@ -364,14 +365,22 @@
                                             <POSTGRES_DB>${keycloak.connectionsJpa.database}</POSTGRES_DB>
                                             <POSTGRES_USER>${keycloak.connectionsJpa.user}</POSTGRES_USER>
                                             <POSTGRES_PASSWORD>${keycloak.connectionsJpa.password}</POSTGRES_PASSWORD>
+
+                                            <!-- MSSQL -->
+                                            <ACCEPT_EULA>Y</ACCEPT_EULA>
+                                            <SA_PASSWORD>${keycloak.connectionsJpa.password}</SA_PASSWORD>
+
+                                            <!-- Oracle -->
+                                            <ORACLE_SID>${keycloak.connectionsJpa.database}</ORACLE_SID>
+                                            <ORACLE_PWD>sa</ORACLE_PWD>
                                         </env>
                                         <wait>
-                                            <tcp>
+                                            <!-- Do not use waiting for port since that is unreliable, sometimes port is listening before DB is ready to serve -->
-                                                <ports>
+                                            <log>${docker.database.wait-for-log-regex}</log>
-                                                    <port>${docker.database.port}</port>
+                                            <time>300000</time>
-                                                </ports>
+                                            <exec>
-                                            </tcp>
+                                              <postStart>${docker.database.postStart}</postStart>
-                                            <time>120000</time>
+                                            </exec>
                                         </wait>
                                     </run>
                                 </image>

testsuite/integration-arquillian/tests/base/src/test/resources/META-INF/keycloak-server.json

@@ -97,10 +97,10 @@
 
     "connectionsJpa": {
         "default": {
-            "url": "${keycloak.connectionsJpa.url:jdbc:h2:mem:test;MVCC=TRUE;DB_CLOSE_DELAY=-1}",
+            "url": "${keycloak.connectionsJpa.url:SET-THE-keycloak.connectionsJpa.URL-PROPERTY}",
-            "driver": "${keycloak.connectionsJpa.driver:org.h2.Driver}",
+            "driver": "${keycloak.connectionsJpa.driver:SET-THE-keycloak.connectionsJpa.driver-PROPERTY}",
             "driverDialect": "${keycloak.connectionsJpa.driverDialect:}",
-            "user": "${keycloak.connectionsJpa.user:sa}",
+            "user": "${keycloak.connectionsJpa.user:}",
             "password": "${keycloak.connectionsJpa.password:}",
             "initializeEmpty": true,
             "migrationStrategy": "update",

testsuite/integration-arquillian/tests/pom.xml

@@ -1236,46 +1236,6 @@
             </build>
         </profile>
 
-        <profile>
-            <id>jdbc-driver-dependency</id>
-            <activation>
-                <property>
-                    <name>jdbc.mvn.artifactId</name>
-                </property>
-            </activation>
-            <dependencies>
-                <dependency>
-                    <groupId>${jdbc.mvn.groupId}</groupId>
-                    <artifactId>${jdbc.mvn.artifactId}</artifactId>
-                    <version>${jdbc.mvn.version}</version>
-                </dependency>
-            </dependencies>
-            <build>
-                <plugins>
-                    <plugin>
-                        <artifactId>maven-enforcer-plugin</artifactId>
-                        <executions>
-                            <execution>
-                                <goals>
-                                    <goal>enforce</goal>
-                                </goals>
-                                <configuration>
-                                    <rules>
-                                        <requireProperty>
-                                            <property>jdbc.mvn.groupId</property>
-                                        </requireProperty>
-                                        <requireProperty>
-                                            <property>jdbc.mvn.version</property>
-                                        </requireProperty>
-                                    </rules>
-                                </configuration>
-                            </execution>
-                        </executions>
-                    </plugin>
-                </plugins>
-            </build>
-        </profile>
-
         <!-- Profiles for migration tests-->
 
         <profile>
@@ -1500,6 +1460,7 @@
                 <dependency>
                     <groupId>com.h2database</groupId>
                     <artifactId>h2</artifactId>
+                    <version>${h2.version}</version>
                     <scope>compile</scope>
                 </dependency>
 
@@ -1628,18 +1589,10 @@
                 </dependency>
 
                 <dependency>
-                    <groupId>mysql</groupId>
+                    <groupId>${jdbc.mvn.groupId}</groupId>
-                    <artifactId>mysql-connector-java</artifactId>
+                    <artifactId>${jdbc.mvn.artifactId}</artifactId>
-                </dependency>
+                    <version>${jdbc.mvn.version}</version>
-                <dependency>
+                    <scope>compile</scope>
-                    <groupId>org.postgresql</groupId>
-                    <artifactId>postgresql</artifactId>
-                    <version>${postgresql.version}</version>
-                </dependency>
-                <dependency>
-                    <groupId>org.mariadb.jdbc</groupId>
-                    <artifactId>mariadb-java-client</artifactId>
-                    <version>${mariadb.version}</version>
                 </dependency>
 
                 <!-- CLI -->