Possibility to test EAP6 cluster with docker+fig

This commit is contained in:
mposolda 2014-10-07 12:33:27 +02:00
parent 1604315e40
commit a6bbc492b8
19 changed files with 294 additions and 51 deletions

View file

@ -3,7 +3,7 @@ How to test Keycloak cluster with Docker
Docker+Fig allows to easily setup and test the whole environment with:
* Apache HTTPD 2.4 + modcluster 1.3 as Load Balancer
* MySQL 5.6.1 as database
* Various number of Keycloak cluster nodes running on WildFly (with "demo" examples deployed)
* Various number of Keycloak cluster nodes running on WildFly with "demo" examples deployed. (See below for EAP 6.3 and AS7)
You don't need to setup Apache with modcluster + MySQL on your laptop as Docker will do it for you and all will run in Docker containers.
@ -42,7 +42,6 @@ be able to access Apache modCluster status page: [http://localhost:10001/mod_clu
with deployed "auth-server.war" and few other WARs (keycloak demo).
Also you can access Keycloak admin console via loadBalancer on [http://localhost:8000/auth/admin](http://localhost:8000/auth/admin) and similarly Account mgmt.
TODO: Examples currently doesn't work and I am looking at it..
MySQL can be directly accessed from your machine (if you have MySQL client installed):
```shell
@ -74,7 +73,7 @@ Scale / more cluster nodes
Run this in separate terminal to add more (in this case 2) cluster nodes:
```shell
$ fig scale node=2
$ fig scale wfnode=2
````
Now it should be visible on mod_cluster_manager page that they are 2 nodes.
@ -89,7 +88,7 @@ to see output of MySql and Keycloak server consoles.
To see Apache and debug logs of keycloak server:
```shell
$ fig run node /bin/bash
$ fig run wfnode /bin/bash
````
Then you're in shell inside docker container, which has some mounted volumes with apache logs and keycloak nodes. Apache logs are at:
@ -133,3 +132,26 @@ In this case you might need to stop and remove existing containers. Then start f
changed jars, then rebuild distribution and testsuite/docker-cluster
(or just copy changed JAR into $KEYCLOAK_HOME/testsuite/docker-cluster/target/keycloak-docker-cluster/deployments/auth-server.war/WEB-INF/lib if it's not adapter stuff.
But 'fig rm' is safer to call anyway)
Test with Keycloak and examples on EAP 6.3
------------------------------------------
Steps are quite similar like for WildFly but we need to pass different file "fig-eap63.yml" instead of default "fig.yml" which is used for WildFly.
Also name of the node is "eapnode" instead of "wfnode".
So your commands will look like
```shell
$ fig -f fig-eap63.yml build
$ fig -f fig-eap63.yml up
$ fig -f fig-eap63.yml scale eapnode=2
````
and viceversa.
Test with Keycloak and examples on AS 7.1.1
-------------------------------------------
Also arguments need to be passed with different fig file and node name: TODO: AS7 cluster setup doesn't work correctly yet
```shell
$ fig -f fig-as7.yml build
$ fig -f fig-as7.yml up
$ fig -f fig-as7.yml scale asnode=2
````

View file

@ -0,0 +1,32 @@
FROM jboss/wildfly
USER root
# Update yum and install required programs
RUN yum install -y unzip && yum install -y wget && yum install -y mc && yum -y install nc
RUN yum clean all
# Download mysql driver
RUN cd /tmp
RUN wget http://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/5.1.32/mysql-connector-java-5.1.32.jar
RUN mv *.jar mysql-connector-java-5.1.32.jar
# Drop wildfly
RUN rm -rf /opt/wildfly*
ENV AS7_VERSION 7.1.1.Final
# Download and unpack AS7 distribution
RUN cd /opt
RUN wget http://download.jboss.org/jbossas/7.1/jboss-as-$AS7_VERSION/jboss-as-$AS7_VERSION.zip
RUN sleep 3
RUN unzip -q jboss-as-$AS7_VERSION.zip
# Make sure the distribution is available from a well-known place
RUN mv jboss-as-$AS7_VERSION /opt/as7
RUN rm -rf jboss-as-$AS7_VERSION.zip
EXPOSE 8787
CMD [ "/bin/bash" ]

View file

@ -0,0 +1,13 @@
FROM mposolda/as7
ADD keycloak-as7-trigger.sh /keycloak-as7-trigger.sh
RUN chmod u+x /keycloak-as7-trigger.sh
ENV JBOSS_HOME /opt/as7
ENV JBOSS_MODULES_HOME $JBOSS_HOME/modules
ENV JBOSS_TYPE as7
ENV NODE_PREFIX as
EXPOSE 8787
CMD [ "/keycloak-as7-trigger.sh" ]

View file

@ -0,0 +1,8 @@
#!/bin/bash
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-run-node.sh
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-base-prepare.sh
chmod u+x /keycloak-docker-cluster/shared-files/deploy-examples.sh
echo "Permissions changed. Triggering keycloak-run-node.sh"
/keycloak-docker-cluster/shared-files/keycloak-run-node.sh

View file

@ -23,8 +23,20 @@
</excludes>
</fileSet>
<fileSet>
<directory>target/modules</directory>
<outputDirectory>modules</outputDirectory>
<directory>target/wildfly-adapter</directory>
<outputDirectory>wildfly-adapter</outputDirectory>
</fileSet>
<fileSet>
<directory>target/as7-adapter</directory>
<outputDirectory>as7-adapter</outputDirectory>
</fileSet>
<fileSet>
<directory>target/eap63-adapter</directory>
<outputDirectory>eap63-adapter</outputDirectory>
</fileSet>
<fileSet>
<directory>shared-files</directory>
<outputDirectory>shared-files</outputDirectory>
</fileSet>
</fileSets>
</assembly>

View file

@ -0,0 +1,30 @@
FROM jboss/wildfly
USER root
# Update yum and install required programs
RUN yum install -y unzip && yum install -y wget && yum install -y mc && yum -y install nc
RUN yum clean all
# Download mysql driver
RUN cd /tmp
RUN wget http://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/5.1.32/mysql-connector-java-5.1.32.jar
RUN mv *.jar mysql-connector-java-5.1.32.jar
# Drop wildfly
RUN rm -rf /opt/wildfly*
# Download and unpack EAP63 distribution TODO: Check if it's an issue for EAP 6.3
RUN cd /
RUN wget https://dl.dropboxusercontent.com/u/5525920/jboss-eap-6.3.0.zip
RUN sleep 3
RUN unzip -q jboss-eap-6.3.0.zip
# Make sure the distribution is available from a well-known place
RUN mv jboss-eap-6.3 /opt/eap63
RUN rm -rf jboss-eap-6.3.0.zip
EXPOSE 8787
CMD [ "/bin/bash" ]

View file

@ -0,0 +1,13 @@
FROM mposolda/eap63
ADD keycloak-eap63-trigger.sh /keycloak-eap63-trigger.sh
RUN chmod u+x /keycloak-eap63-trigger.sh
ENV JBOSS_HOME /opt/eap63
ENV JBOSS_MODULES_HOME $JBOSS_HOME/modules/system/layers/base
ENV JBOSS_TYPE eap63
ENV NODE_PREFIX eap
EXPOSE 8787
CMD [ "/keycloak-eap63-trigger.sh" ]

View file

@ -0,0 +1,8 @@
#!/bin/bash
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-run-node.sh
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-base-prepare.sh
chmod u+x /keycloak-docker-cluster/shared-files/deploy-examples.sh
echo "Permissions changed. Triggering keycloak-run-node.sh"
/keycloak-docker-cluster/shared-files/keycloak-run-node.sh

View file

@ -0,0 +1,31 @@
httpd:
build: httpd
ports:
- "8000:80"
- "10001:10001"
volumes_from:
- mysql
mysql:
image: mysql:5.6.20
environment:
- MYSQL_ROOT_PASSWORD=mysecretpassword
- MYSQL_DATABASE=keycloak_db
volumes:
- /keycloak-docker-shared
- /apachelogs
ports:
- "33306:3306"
asnode:
build: as7
command: /keycloak-as7-trigger.sh
volumes:
- target/keycloak-docker-cluster:/keycloak-docker-cluster
volumes_from:
- mysql
links:
- httpd:httpd
- mysql:mysql
ports:
- "8787"
- "8080"
- "9990"

View file

@ -0,0 +1,31 @@
httpd:
build: httpd
ports:
- "8000:80"
- "10001:10001"
volumes_from:
- mysql
mysql:
image: mysql:5.6.20
environment:
- MYSQL_ROOT_PASSWORD=mysecretpassword
- MYSQL_DATABASE=keycloak_db
volumes:
- /keycloak-docker-shared
- /apachelogs
ports:
- "33306:3306"
eapnode:
build: eap63
command: /keycloak-eap63-trigger.sh
volumes:
- target/keycloak-docker-cluster:/keycloak-docker-cluster
volumes_from:
- mysql
links:
- httpd:httpd
- mysql:mysql
ports:
- "8787"
- "8080"
- "9990"

View file

@ -15,9 +15,9 @@ mysql:
- /apachelogs
ports:
- "33306:3306"
node:
wfnode:
build: wildfly
command: /keycloak-run-node.sh
command: /keycloak-wildfly-trigger.sh
volumes:
- target/keycloak-docker-cluster:/keycloak-docker-cluster
volumes_from:

View file

@ -51,7 +51,21 @@
<artifactId>keycloak-wildfly-adapter-dist</artifactId>
<version>${project.version}</version>
<type>zip</type>
<outputDirectory>${project.build.directory}</outputDirectory>
<outputDirectory>${project.build.directory}/wildfly-adapter</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.keycloak</groupId>
<artifactId>keycloak-as7-adapter-dist</artifactId>
<version>${project.version}</version>
<type>zip</type>
<outputDirectory>${project.build.directory}/as7-adapter</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.keycloak</groupId>
<artifactId>keycloak-eap6-adapter-dist</artifactId>
<version>${project.version}</version>
<type>zip</type>
<outputDirectory>${project.build.directory}/eap63-adapter</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>

View file

@ -4,10 +4,10 @@
# Deploy examples
cd /keycloak-docker-cluster/examples
for I in $(find . | grep .war$); do cp $I /opt/wildfly/standalone/deployments/; done;
for I in $(find . | grep .war$); do cp $I $JBOSS_HOME/standalone/deployments/; done;
# Explode wars
cd /opt/wildfly/standalone/deployments/
cd $JBOSS_HOME/standalone/deployments/
for I in $(ls -d *.war | grep -v auth-server.war); do
echo "Configuring $I";
mkdir $I.tmp;

View file

@ -0,0 +1,29 @@
#!/bin/bash
# Copy MySQL driver
cd /tmp
mkdir -p mysql/main && mv /mysql-connector-java-5.1.32.jar mysql/main/
cp /keycloak-docker-cluster/shared-files/mysql-module.xml mysql/main/module.xml
mv mysql $JBOSS_MODULES_HOME/com/
if [ $JBOSS_TYPE == "eap63" ]; then
EXT="as7";
else
EXT=$JBOSS_TYPE;
fi;
sed -i -e "s/<extensions>/&\n <extension module=\"org.keycloak.keycloak-$EXT-subsystem\"\/>/" $JBOSS_HOME/standalone/configuration/standalone-ha.xml
sed -i -e 's/<profile>/&\n <subsystem xmlns="urn:jboss:domain:keycloak:1.0"\/>/' $JBOSS_HOME/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<security-domains>/&\n <security-domain name="keycloak">\n <authentication>\n <login-module code="org.keycloak.adapters.jboss.KeycloakLoginModule" flag="required"\/>\n <\/authentication>\n <\/security-domain>/' $JBOSS_HOME/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<drivers>/&\n <driver name="mysql" module="com.mysql">\n <xa-datasource-class>com.mysql.jdbc.Driver<\/xa-datasource-class>\n <driver-class>com.mysql.jdbc.Driver<\/driver-class>\n <\/driver>/' $JBOSS_HOME/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<\/periodic-rotating-file-handler>/&\n <logger category=\"org.keycloak\">\n <level name=\"DEBUG\" \/> \n <\/logger>\n <logger category=\"org.jboss.resteasy.core.ResourceLocator\">\n <level name=\"ERROR\" \/> \n <\/logger>/' $JBOSS_HOME/standalone/configuration/standalone-ha.xml
sed -i -e 's/<subsystem xmlns=\"urn:jboss:domain:infinispan:[0-9]\.[0-9]\">/&\n <cache-container name=\"keycloak\" jndi-name=\"infinispan\/Keycloak\" start=\"EAGER\"> \
\n <transport lock-timeout=\"60000\"\/>\n <distributed-cache name=\"sessions\" mode=\"SYNC\" owners=\"2\" segments=\"60\"\/> \
\n <invalidation-cache name=\"realms\" mode=\"SYNC\"\/>\n <\/cache-container>/' $JBOSS_HOME/standalone/configuration/standalone-ha.xml
sed -i "s|<mod-cluster-config .*>|<mod-cluster-config advertise-socket=\"modcluster\" proxy-list=\"\$\{httpd.proxyList\}\" proxy-url=\"\/\" balancer=\"mycluster\" advertise=\"false\" connector=\"ajp\" sticky-session=\"true\">|" $JBOSS_HOME/standalone/configuration/standalone-ha.xml
sed -i "s|#JAVA_OPTS=\"\$JAVA_OPTS -agentlib:jdwp=transport=dt_socket|JAVA_OPTS=\"\$JAVA_OPTS -agentlib:jdwp=transport=dt_socket|" $JBOSS_HOME/bin/standalone.conf
cp /keycloak-docker-cluster/shared-files/mysql-keycloak-ds.xml $JBOSS_HOME/standalone/deployments/

View file

@ -1,47 +1,51 @@
#!/bin/bash
export MYHOST=node$(echo $MYSQL_NAME | awk -F"/dockercluster[^0-9]*|\/mysql" '{print $2 }');
export MYHOST="$NODE_PREFIX"node$(echo $MYSQL_NAME | awk -F"/dockercluster[^0-9]*|\/mysql" '{print $2 }');
echo "MYHOST is $MYHOST. MYSQL_NAME is $MYSQL_NAME";
function prepareHost
{
if [ -d /keycloak-docker-shared/keycloak-wildfly-$MYHOST ]; then
if [ -d /keycloak-docker-shared/keycloak-$JBOSS_TYPE-$MYHOST ]; then
echo "Node $MYHOST already prepared. Skiping";
return;
fi
echo "Creating keycloak-wildfly-$MYHOST";
echo "Creating keycloak-$JBOSS_TYPE-$MYHOST";
cd /opt/wildfly
cp -r /keycloak-docker-cluster/modules ./
/keycloak-docker-cluster/shared-files/keycloak-base-prepare.sh
echo "Base prepare finished";
cd $JBOSS_HOME
cp -r /keycloak-docker-cluster/$JBOSS_TYPE-adapter/modules ./
# Deploy keycloak
cp -r /keycloak-docker-cluster/deployments/* /opt/wildfly/standalone/deployments/
cp -r /keycloak-docker-cluster/deployments/* $JBOSS_HOME/standalone/deployments/
# Enable Infinispan provider
sed -i "s|keycloak.userSessions.provider:mem|keycloak.userSessions.provider:infinispan|" /opt/wildfly/standalone/deployments/auth-server.war/WEB-INF/classes/META-INF/keycloak-server.json
sed -i "s|keycloak.userSessions.provider:mem|keycloak.userSessions.provider:infinispan|" $JBOSS_HOME/standalone/deployments/auth-server.war/WEB-INF/classes/META-INF/keycloak-server.json
# Deploy and configure examples
/deploy-examples.sh
/keycloak-docker-cluster/shared-files/deploy-examples.sh
# Deploy to volume
rm -rf /keycloak-docker-shared/keycloak-wildfly-$MYHOST
cp -r /opt/wildfly-8.1.0.Final /keycloak-docker-shared/keycloak-wildfly-$MYHOST
chmod -R 777 /keycloak-docker-shared/keycloak-wildfly-$MYHOST
echo "keycloak-wildfly-$MYHOST prepared and copyied to volume";
rm -rf /keycloak-docker-shared/keycloak-$JBOSS_TYPE-$MYHOST
cp -r $JBOSS_HOME /keycloak-docker-shared/keycloak-$JBOSS_TYPE-$MYHOST
chmod -R 777 /keycloak-docker-shared/keycloak-$JBOSS_TYPE-$MYHOST
echo "keycloak-$JBOSS_TYPE-$MYHOST prepared and copyied to volume";
}
function waitForPreviousNodeStart
{
myHostNumber=$(echo $MYHOST | awk -F"node" '{ print $2 }');
if [ $myHostNumber -eq 1 ]; then
echo "Our host is node1. No need to wait for previous server";
echo "Our host is $MYHOST. No need to wait for previous server";
else
previous=node$(($myHostNumber-1));
previous="$NODE_PREFIX"node$(($myHostNumber-1));
echo "Waiting for host $previous to start";
for I in $(seq 1 10); do
cat /keycloak-docker-shared/keycloak-wildfly-$previous/standalone/log/server.log | grep "\(INFO\|ERROR\).*WildFly.*started";
cat /keycloak-docker-shared/keycloak-$JBOSS_TYPE-$previous/standalone/log/server.log | grep "\(INFO\|ERROR\).*\(WildFly\|JBoss AS\|JBoss EAP\).*started";
if [ 0 -eq $? ]; then
echo "Host $previous started. Going to start $MYHOST";
return;
@ -77,7 +81,7 @@ waitForMySQLStart;
echo "Running keycloak node $MYHOST. Additional arguments: $@";
cd /keycloak-docker-shared
export JBOSS_HOME=/keycloak-docker-shared/keycloak-wildfly-$MYHOST;
export JBOSS_HOME=/keycloak-docker-shared/keycloak-$JBOSS_TYPE-$MYHOST;
cd $JBOSS_HOME/bin/

View file

@ -1,37 +1,25 @@
FROM jboss/wildfly
USER root
# Update yum and install required programs
RUN yum install -y unzip && yum install -y wget && yum install -y mc && yum -y install nc
RUN yum clean all
# Download mysql driver
RUN cd /tmp
RUN wget http://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/5.1.32/mysql-connector-java-5.1.32.jar
RUN mv *.jar mysql-connector-java-5.1.32.jar
RUN mv *.jar /mysql-connector-java-5.1.32.jar
RUN mkdir -p mysql/main && mv mysql-connector-java-5.1.32.jar mysql/main/
ADD mysql-module.xml mysql/main/module.xml
RUN mv mysql /opt/wildfly/modules/system/layers/base/com/
ADD keycloak-wildfly-trigger.sh /keycloak-wildfly-trigger.sh
RUN chmod u+x /keycloak-wildfly-trigger.sh
RUN sed -i -e 's/<extensions>/&\n <extension module="org.keycloak.keycloak-wildfly-subsystem"\/>/' /opt/wildfly/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<profile>/&\n <subsystem xmlns="urn:jboss:domain:keycloak:1.0"\/>/' /opt/wildfly/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<security-domains>/&\n <security-domain name="keycloak">\n <authentication>\n <login-module code="org.keycloak.adapters.jboss.KeycloakLoginModule" flag="required"\/>\n <\/authentication>\n <\/security-domain>/' /opt/wildfly/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<drivers>/&\n <driver name="mysql" module="com.mysql">\n <xa-datasource-class>com.mysql.jdbc.Driver<\/xa-datasource-class>\n <\/driver>/' /opt/wildfly/standalone/configuration/standalone-ha.xml && \
sed -i -e 's/<\/periodic-rotating-file-handler>/&\n <logger category=\"org.keycloak\">\n <level name=\"DEBUG\" \/> \n <\/logger>/' /opt/wildfly/standalone/configuration/standalone-ha.xml
RUN sed -i -e 's/<subsystem xmlns=\"urn:jboss:domain:infinispan:2\.0\">/&\n <cache-container name=\"keycloak\" jndi-name=\"infinispan\/Keycloak\" start=\"EAGER\"> \
\n <transport lock-timeout=\"60000\"\/>\n <distributed-cache name=\"sessions\" mode=\"SYNC\" owners=\"2\" segments=\"60\"\/> \
\n <invalidation-cache name=\"realms\" mode=\"SYNC\"\/>\n <\/cache-container>/' /opt/wildfly/standalone/configuration/standalone-ha.xml
RUN sed -i "s|<mod-cluster-config .*>|<mod-cluster-config advertise-socket=\"modcluster\" proxy-list=\"\$\{httpd.proxyList\}\" proxy-url=\"\/\" balancer=\"mycluster\" advertise=\"false\" connector=\"ajp\" sticky-session=\"true\">|" /opt/wildfly/standalone/configuration/standalone-ha.xml
RUN sed -i "s|#JAVA_OPTS=\"\$JAVA_OPTS -agentlib:jdwp=transport=dt_socket|JAVA_OPTS=\"\$JAVA_OPTS -agentlib:jdwp=transport=dt_socket|" /opt/wildfly/bin/standalone.conf
ADD mysql-keycloak-ds.xml /opt/wildfly/standalone/deployments/
ADD keycloak-run-node.sh /keycloak-run-node.sh
RUN chmod u+x /keycloak-run-node.sh
ADD deploy-examples.sh /deploy-examples.sh
RUN chmod u+x /deploy-examples.sh
ENV JBOSS_HOME /opt/wildfly-8.1.0.Final
ENV JBOSS_MODULES_HOME $JBOSS_HOME/modules/system/layers/base
ENV JBOSS_TYPE wildfly
ENV NODE_PREFIX wf
EXPOSE 8787
CMD [ "/keycloak-run-node.sh" ]
CMD [ "/keycloak-wildfly-trigger.sh" ]

View file

@ -0,0 +1,8 @@
#!/bin/bash
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-run-node.sh
chmod u+x /keycloak-docker-cluster/shared-files/keycloak-base-prepare.sh
chmod u+x /keycloak-docker-cluster/shared-files/deploy-examples.sh
echo "Permissions changed. Triggering keycloak-run-node.sh"
/keycloak-docker-cluster/shared-files/keycloak-run-node.sh