diff --git a/testsuite/integration-arquillian/README.md b/testsuite/integration-arquillian/README.md index 7620d980f4..36a8252533 100644 --- a/testsuite/integration-arquillian/README.md +++ b/testsuite/integration-arquillian/README.md @@ -1,61 +1,155 @@ -# Keycloak Integration Testsuite with Arquillian +# Keycloak Arquillian Integration Testsuite -## Structure +## Container Lifecycles + +### Keycloak Auth Server + +There is only one instance of Keycloak server running during a single test run. +It is automatically started by Arquillian on the `BeforeSuite` event and stopped `AfterSuite`. + +The type of container can be determined by property `-Dauth.server.container`. Default value is `auth-server-undertow`, +other options are: `auth-server-wildfly` and `auth-server-eap7`. The values correspond to Arquillian *container qualifiers* in `arquillian.xml` config file. + +**Note 1:** For the non-default options it's necessary to build a corresponding server module prior to running any of the test modules. +This can be done by building the server module directly (from `servers/wildfly`/`servers/eap7`), +or by activating `auth-server-wildfly`/`auth-server-eap7` profile when building from the top level module. +(The profiles will also set the proper value of the `auth.server.container` property.) + +**Note 2:** Most server-side configurations are done during the build of the server module +and included in the output artifact - which is then consumed by the test modules( if a corresponding profile is activated). +To reflect a change in server config in the test (e.g. a datasource) it's necessary to rebuild the server module after each change. + +### Migration + +Migration tests can be enabled by setting `-Dmigrated.auth.server.container` property or activating a corresponding profile. +When enabled, the `AuthServerTestEnricher` class will start/stop the selected *migrated* instance +even **before** the *current* auth server instance is started. + +### App Servers + +Lifecycle of application server is always tied to a particular TestClass. + +Each *adapter* test class is annotated by `@AppServerContainer("app-server-*")` annotation +that links it to a particular Arquillian container in `arquillian.xml`. +The `AppServerTestEnricher` then ensures the server is started before and stopped after all tests methods in the class. +In case the `@AppServerContainer` annotation has no value it's assumed that the application container +is the same as the auth server container (a relative adapter test scenario). + +Adapter tests are separated into submodules because different app containers require different configurations +(installation of adapter libs, etc.). +Container entries of app servers are not present in the main `arquillian.xml` in the `base` module. +Each adapter submodule adds it's own entry before it runs the tests. + +## SuiteContext and TestContext + +These objects are injected into `AbstractKeycloakTest` class so they can be used everywhere. +They can be used to get information about the tested containers, and to store information that won't survive across test classes or test methods. +(Arquillian creates a new instance of test class for each test method run, so all data in the fields is always lost.) + +## REST Testing + +The `AbstractKeycloakTest` has an initialized instance of AdminClient. Subclasses can use it to access any REST subresources. + +## UI Testing + +PageObjects can be injected to tests by `@Page` annotation. + +The base class for all page objects is `AbstractPage`. + +The default browser for UI testing is `phantomjs` which is used for fast "headless" testing. +Other browsers can be selected with the `-Dbrowser` property, for example `firefox`. +See *Arquillian Graphene* documentation for more details. + + +## Test Modules + +### Base Testsuite + +The base testsuite contains custom Arquillian extensions and most functional tests. +The other test modules depend on this module. + +### Admin Console UI Tests + +Tests for Keycloak Admin Console are located in a separate module `tests/other/console` +and are **disabled** by default. Can be enabled by `-Pconsole-ui-tests`. + +### Adapter Tests + +Adapter tests are located in submodules of the `tests/other/adapters` module. + +They are **disabled** by default; they can be enabled by corresponding profiles. +Multiple profiles can be enabled for a single test execution. + +#### Types of adapter tests + +1. Using *custom test servlets* +2. Using *example demo apps* from `keycloak/examples` modules. + +#### Relative vs Non-relative scenario + +The test suite can handle both types. +It automatically modifies imported test realms and deployments' adapter configs based on scenario type. + +| Scenario | Description | Realm config (server side) | Adapter config (client side) | +| --- | --- | --- | --- | +| **Relative** | auth server == app server | client `baseUrl`, `adminUrl` and `redirect-uris` can be relative | `auth-server-url` can be relative | +| **Non-relative** | auth server != app server | client `baseUrl`, `adminUrl` and `redirect-uris` need to include FQDN of the app server | `auth-server-url` needs to include FQDN of the auth server| + + + +#### Adapter Libs Mode + +1. **Provided** - By container, e.g. as a subsystem. **Default.** +2. **Bundled** - In the deployed war in `/WEB-INF/libs`. Enable with `-Dadapter.libs.bundled`. *Wildfly only*. + +#### Adapter Config Mode + +1. ~~**Provided** - In `standalone.xml` using `secure-deployment`. *Wildfly only.*~~ WIP +2. **Bundled** - In the deployed war in `/WEB-INF/keycloak.json`. **Default.** + + +## Maven Modules Structure ``` integration-arquillian │ -├──servers (submodules enabled via profiles) -│ ├──wildfly -│ └──eap6 +├──servers +│ ├──wildfly (activated by -Pauth-server-wildfly) +│ └──eap7 (activated by -Pauth-server-eap7) │ -└──tests - ├──base - └──adapters (submodules enabled via profiles, all depend on base) - ├──wildfly - ├──wildfly-relative (needs servers/wildfly) - ├──wildfly8 - ├──as7 - ├──tomcat - └──karaf - +└──tests (common settings for all test modules) + ├──base (custom ARQ extensions + base functional tests) + └──other (common settings for all modules dependent on base) + │ + ├──adapters (common settings for all adapter submodules) + │ ├──wildfly (activated by -Papp-server-wildfly) + │ ├──wildfly-relative (activated by -Papp-server-wildfly-relative,auth-server-wildfly) + │ ├──... + │ ├──tomcat (activated by -Papp-server-tomcat) + │ └──karaf (activated by -Papp-server-karaf) + │ + ├──console (activated by -Pconsole-ui-tests) + ├──mod_auth_mellon (activated by -Pmod_auth_mellon) + └──... ``` -## General Concepts +## Custom Arquillian Extensions -The testsuite supports **multiple server runtimes** for the Keycloak server. -The **default is Undertow** which is the fastest and easiest option, and runs in the same JVM as the tests. +Custom extensions are registered in `META-INF/services/org.jboss.arquillian.core.spi.LoadableExtension`. -Other options are **Wildfly 9** and **EAP 6**. These have some additional requirements and limitations: -1. The selected server module must be built before any tests can be run. -All server-side configuration is done during this build (e.g. datasource configuration). -Once server artifact is built the tests modules can unpack it via `maven-dependency-plugin` into their working directory before running. -2. Before the selected server module can be built the `keycloak/distribution` module also needs to be built. +### MultipleContainersExtension + * Replaces Arquillian's default container handling. + * Allows to manage multiple container instances of different types within a single test run. + * Allows to skip loading disabled containers based on `enabled` config property in `arquillian.xml`. -### Server Runtimes +### KeycloakArquillianExtension + * `AuthServerTestEnricher` - Handles lifecycle of auth server and migrated auth server. + * `AppServerTestEnricher` - Handles lifecycles of app servers. + * `CustomUndertowContainer` - Custom container controller for JAX-RS-enabled Undertow with Keycloak Server. + * `DeploymentArchiveProcessor` - Modifies adapter configs before test apps are deployed. + * `DeploymentTargetModifier` - Ensures all test app deployments are targeted at app server containers. + * `URLProvider` - Fixes URLs injected by Arquillian Graphene which contain value `127.0.0.1` instead of required `localhost`. + +### CustomKarafContainerExtension -TODO: explain why separate module, list config options, note on migration modules - -### Base Testsuite - -login flows + account management - -admin ui - -abstract adapter tests - -### Adapter Tests - -test servlets: demo, session - -examples - -## Running the Tests - -### Undertow - -### Wildfly or EAP 6 - -### Adapters - -### Supported Browsers \ No newline at end of file +Extension for executing karaf commands after container is started. Used for installation of bundles (test apps and adapter libs). \ No newline at end of file diff --git a/testsuite/integration-arquillian/README_old.md b/testsuite/integration-arquillian/README_old.md deleted file mode 100644 index 76357ce984..0000000000 --- a/testsuite/integration-arquillian/README_old.md +++ /dev/null @@ -1,189 +0,0 @@ -# Keycloak Integration Testsuite with Arquillian - -*OUT OF DATE - NEEDS REWRITE* - -## Usage - -Running the tests: `mvn test` or `mvn clean test` - -## Test suite - -### Selecting container for Keycloak Server - -The testsuite requires a container for Keycloak Server to be selected. -This container is used by all tests in the suite during a single test execution. - -*By default* the tests run with server on embedded *Undertow*. -A different container can be selected with profile, e.g. `-Pauth-server-wildfly`. - -### Containers Supported for Keycloak Server - -| Container | Arquillian Qualifier | Maven | Dependencies | -| --- | --- | --- | --- | -| **Undertow** | `auth-server-undertow` | **default** | `undertow-core`, `resteasy-undertow` | -| **Wildfly 9** | `auth-server-wildfly` | `-Pauth-server-wildfly` | `keycloak-server-dist` or `wildfly-dist`+`keycloak-server-overlay` | -| **EAP 6.4** | `auth-server-eap6` | `-Pauth-server-eap6` | `keycloak-server-dist` or `eap6-dist`+`keycloak-server-overlay` | - -See the relevant container definitions in `arquillian.xml` located in the **test resources** folder. - -### Test Class Hierarchy -``` -AbstractKeycloakTest -├── AbstractAdminConsoleTest -└── AbstractAdapterTest -``` - -### AbstractKeycloakTest - -Handles test realms. Provides Admin Client for REST operations. - -* **@BeforeClass** - 1. Updates the admin password to enable the admin user. -* **@Before** - 1. Initiates admin client - 2. Imports test realms. (Loading test realms is overriden in subclasses.) -* **@After** - 1. Removes test realms. - 2. Closes admin client. - -### ContainersTestEnricher - -Manages *container lifecycles*. - -`ContainersTestEnricher` is a custom Arquillian observer that handles lifecycles of auth server and app server containers for each test class. -Containers are started during `@BeforeClass` and shut down during `@AfterClass` event. - -*Optionally* each test class can be annotated with `@AuthServerContainer("qualifier")` and `@AppServerConatiner("qualifier")` annotations -to indicate containers required for the test. - -* In case `@AuthServerContainer` is not present the *auth server qualifier* is loaded from `auth.server.container` property. -* In case `@AppServerContainer` is not present or it's value is the same as *auth server qualifier*, the app server isn't started for the test class. - -## Admin Console Tests - -Tests for admin console are located in `org/keycloak/testsuite/console`. -Related non-test classes are located on the same path in the **main sources**. - -Admin console tests are **ENABLED by default**. They can be disabled by `-P no-console`. - - -## Adapter Tests - -Adapter tests are located in `org/keycloak/testsuite/adapter`. -Related non-test classes can be found on the same path in the **main sources**. - -Adapter tests are **DISABLED by default**. They can be enabled by profiles. -Multiple profiles can be enabled for a single test execution. - -*Note:* When testing adapter with multiple containers in a single run it is better -to use the `--fail-at-end` (`-fae`) strategy instead of the default `--fail-fast` one. -This will allow Maven to continue building other modules even if some of them have test failures. - -### Containers Supported for Adapter Tests - -| Container | Arquillian Qualifier | Maven | Dependencies | -| --- | --- | --- | --- | -| **Wildfly 9** Relative | `auth-server-wildfly` | `-Pauth-server-wildfly` | `keycloak-server-dist` or `wildfly-dist`+`keycloak-server-overlay`, `keycloak-adapter-dist-wf9` | -| **Wildfly 9** | `app-server-wildfly` | `-Papp-server-wildfly` | `wildfly-dist`, `keycloak-adapter-dist-wf9` | -| **Wildfly 8** | `app-server-wildfly` | `-Papp-server-wildfly8` | `wildfly-dist:8.2.1.Final`, `keycloak-adapter-dist-wf8` | -| **JBoss AS 7** | `app-server-as7` | `-Papp-server-as7` | `jboss-as-dist`, `keycloak-adapter-dist-as7` | -| **Tomcat 8** | `app-server-tomcat` | `-Papp-server-tomcat` | `tomcat`, `keycloak-tomcat8-adapter-dist` | -| **Karaf 3** | `app-server-karaf` | `-Papp-server-karaf` | `apache-camel`, `apache-cxf`, `keycloak-osgi-features`, `keycloak-fuse-example-features` | - -See the relevant container definitions in `tests/adapter//src/main/xslt/arquillian.xsl`. - -***Important:*** Arquillian cannot load multiple controllers for JBossAS/Wildfly containers in a single run (because same class name) -but a different controller is required for JBossAS7/EAP6 than for WF8/9. Because of this: - - - Adapter tests for *Wildfly 8/9* cannot be run against server on *EAP 6*. `-Papp-server-wildfly*` ⇒ `!auth-server-eap6` - - Adapter tests for *JBossAS 7* can only be run against server on *EAP 6*. `-Papp-server-as7,auth-server-eap6` - -### Adapter Test Types - -1. Using **test servlets**. -2. Using **example/demo wars**. - -``` -AbstractKeycloakTest -└── AbstractAdapterTest - ├── AbstractServletsAdapterTest - | ├── Relative… - | ├── Wildfly… - | ├── Tomcat… - | … - └── AbstractExampleAdapterTest - ├── AbstractDemoExampleAdapterTest - | ├── Relative… - | ├── Wildfly… - | ├── Tomcat… - | … - ├── AbstractBasicAuthExampleAdapterTest - | ├── Relative… - | ├── Wildfly… - | ├── Tomcat… - | … - … -``` - -### Relative vs Non-relative scenario - -The test suite can handle both types. -It automatically modifies imported test realms and deployments' adapter configs based on scenario type. - -| Scenario | Description | Realm config (server side) | Adapter config (client side) | -| --- | --- | --- | --- | -| **Relative** | Both Keycloak Server and test apps running in the same container. | client `baseUrl`, `adminUrl` and `redirect-uris` can be relative | `auth-server-url` can be relative | -| **Non-relative** | Test apps run in a different container than Keycloak Server. | client `baseUrl`, `adminUrl` and `redirect-uris` need to include FQDN of the app server | `auth-server-url` needs to include FQDN of the auth server| - -### Adapter Libraries Mode - -1. **Provided.** By container, e.g. as a subsystem. *Default.* -2. **Bundled.** In the deployed war in `/WEB-INF/libs`. Enable with `-Dadapter.libs.bundled`. *Wildfly only*. - -### Adapter Config Mode - -1. ~~**Provided.** In `standalone.xml` using `secure-deployment`. *Wildfly only.*~~ WIP -2. **Bundled.** In the deployed war in `/WEB-INF/keycloak.json`. *Default.* - -### Adapters Test Coverage - -| Module | Coverage | Supported Containers | -| --- | --- | --- | -| ***Test Servlets*** | Good | All | -| **Demo** | Minimal, WIP | `auth-server-wildfly` (relative) | -| **Admin Client** | | -| **Cordova** | | -| **CORS** | | -| **JS Console** | Good | `auth-server-wildfly` (relative) | -| **Providers** | | -| Themes | | -| Multitenancy | WIP | | -| **Basic Auth** | Good | All | -| **Fuse** | Good | `app-server-karaf` | -| SAML | | -| LDAP | | -| Kerberos | | - -## Supported Browsers - -| Browser | Maven | -| --- | --- | -| **PhantomJS** | `-Dbrowser=phantomjs` **default** | -| **Firefox** | `-Dbrowser=firefox` | - - -## Custom Arquillian Extensions - -Custom extensions are registered in `META-INF/services/org.jboss.arquillian.core.spi.LoadableExtension`. - -* Multiple containers extension - * Replaces Arquillian's default container handling. - * Allows to manage multiple container instances of different types within a single test run. - * Allows to skip loading disabled containers based on `enabled` config property in `arquillian.xml`. -* Custom extension - * `ContainersTestEnricher` - Handles lifecycles of auth-server and app-server. - * `CustomUndertowContainer` - A custom container controller for JAX-RS-enabled Undertow with Keycloak Server. - * `DeploymentArchiveProcessor` - Modifies adapter config before deployment on app server based on relative/non-relative scenario. - * `URLProvider` - Fixes URLs injected by Arquillian which contain 127.0.0.1 instead of localhost. - * `JiraTestExecutionDecider` - Skipping tests for unresolved JIRAs. - diff --git a/testsuite/integration-arquillian/tests/base/src/test/resources/arquillian.xml b/testsuite/integration-arquillian/tests/base/src/test/resources/arquillian.xml index 57e1692dcd..555f3ce35a 100644 --- a/testsuite/integration-arquillian/tests/base/src/test/resources/arquillian.xml +++ b/testsuite/integration-arquillian/tests/base/src/test/resources/arquillian.xml @@ -181,5 +181,12 @@ +