keycloak-scim/testsuite/performance/README.datasets.md

# Keycloak Datasets

## Provision Keycloak Server

Before generating data it is necessary to provision/start Keycloak server. This can 
be done automatically by running:

```
cd testsuite/performance
mvn clean install
mvn verify -P provision
```
To tear down the system after testing run:
```
mvn verify -P teardown
```
The teardown step will delete the database as well so it is possible to use it between generating different datasets.

It is also possible to start the server externally (manually). In that case it is necessary
to provide information in file `tests/target/provisioned-system.properties`.
See the main README for details.

## Generate Data

To generate the *default dataset* run: 
```
cd testsuite/performance
mvn verify -P generate-data
```

To generate a *specific dataset* from within the project run:
```
mvn verify -P generate-data -Ddataset=<NAMED_DATASET>
```
This will load dataset properties from `tests/src/test/resources/dataset/${dataset}.properties`.

To generate a specific dataset from a *custom properties file* run:
```
mvn verify -P generate-data -Ddataset.properties.file=<FULL_PATH_TO_PROPERTIES_FILE>
```

To delete a dataset run:
```
mvn verify -P generate-data -Ddataset=… -Ddelete=true
```
This will delete all realms specified by the dataset.


## Indexed Model

The model is hierarchical with the parent-child relationships determined by primary foreign keys of entities.

Size of the dataset is determined by specifying a "count per parent" parameter for each entity.

Number of mappings between entities created by the primary "count per parent" parameters
can be speicied by "count per other entity" parameters.

Each nested entity has a unique index which identifies it inside its parent entity.

For example:
- Realm X --> Client Y --> Client Role Z
- Realm X --> Client Y --> Resource Server --> Resource Z
- Realm X --> User Y
- etc.

Hash code of each entity is computed based on its index coordinates within the model and its class name.

Each entity holds entity representation, and a list of mappings to other entities in the indexed model.
The attributes and mappings are initialized by a related *entity template* class.
Each entity class also acts as a wrapper around a Keycloak Admin Client using it 
to provide CRUD operations for its entity.

The `id` attribute in the entity representation is set upon entity creation, or in case 
an already initialized entity was removed from LRU cache it is reloaded from the server.
This may happen if the number of entities is larger than entity cache size. (see below)

### Attribute Templating

Attributes of each supported entity representation can be set via FreeMarker templates.
The process is based on templates defined in a properties configuration file.

The first template in the list can use the `index` of the entity and any attributes of its parent entity.
Each subsequent attribute template can use any previously set attribute values.

Note: Output of FreeMarker engine is always a String. Transition to the actual type 
of the attribute is done with the Jackson 2.9+ parser using `ObjectMapper.update()` method
which allows a gradual updates of an existing Java object.

### Randomness

Randomness in the indexed model is deterministic (pseudorandom) because the 
random seeds are based on deterministic hash codes.

There are 2 types of seeds: one is for using randoms in the FreeMarker templates 
via methods `indexBasedRandomInt(int bound)` and `indexBasedRandomBool(int percentage)`.
It is based on class of the current entity + hash code of its parent entity.

The other seed is for generating mappings to other entities which are just 
random sequences of integer indexes. This is based on hash code of the current entity.

### Generator Settings

#### Timeouts
- `queue.timeout`: How long to wait for an entity to be processed by a thread-pool executor. Default is `60` seconds. 
You might want to increase this setting when deleting many realms with many nested entities using a low number of workers.
- `shutdown.timeout`: How long to wait for the executor thread-pool to shut down. Default is `60` seconds.

#### Caching and Memory
- `template.cache.size`: Size of cache of FreeMarker template models. Default is `10000`.
- `randoms.cache.size`: Size of cache of random integer sequences which are used for mappings between entities. Default is `10000`.
- `entity.cache.size`: Size of cache of initialized entities. Default is `100000`.
- `max.heap`: Max heap size of the data generator JVM.


## Notes:

- Mappings are random so it can sometimes happen that the same mappings are generated multiple times.
Only distinct mappings are created.
This means for example that if you specify `realmRolesPerUser=5` it can happen 
that only 4 or less roles will be actually mapped.

    There is an option to use unique random sequences but is is disabled right now 
because checking for uniqueness is CPU-intensive.

- Mapping of client roles to a user right now is determined by a single parameter: `clientRolesPerUser`. 

    Actually created mappings -- each of which contains specific client + a set of its roles -- is created 
based on the list of randomly selected client roles of all clients in the realm. 
This means the count of the actual client mappings isn't predictable. 

    That would require specifying 2 parameters: `clientsPerUser` and `clientRolesPerClientPerUser`
which would say how many clients a user has roles assigned from, and the number of roles per each of these clients.

- Number of resource servers depends on how the attribute `authorizationServicesEnabled` 
is set for each client. This means the number isn't specified  by any "perRealm" parameter. 
If this is needed it can be implemented via a random mapping from a resource server entity 
to a set of existing clients in a similar fashion to how a resource is selected for each resource permission.

- The "resource type" attribute for each resource and resource-based permission defaults to 
the default type of the parent resource server. 
If it's needed a separate abstract/non-persistable entity ResourceType can be created in the model
to represent a set of resource types. The "resource type" attributes can then be set based on random mappings into this set.

- Generating large number of users can take a long time with the default realm settings
which have the password hashing iterations set to a default value of 27500.
If you wish to speed this process up decrease the value of `hashIterations()` in attribute `realm.passwordPolicy`.

    Note that this will also significantly affect the performance results of the tests because 
password hashing takes a major part of the server's compute resources. The results may
improve even by a factor of 10 or higher when the hashing is set to the minimum value of 1 itreration.
However it's on the expense of security.