Allow migration of non-persistent sessions to persistent sessions

Closes #29375

Signed-off-by: Alexander Schwartz <aschwart@redhat.com>

docs/documentation/release_notes/topics/25_0_0.adoc

@@ -63,22 +63,7 @@ bin/kc.sh build --features=persistent-user-session ...
 For more details see the https://www.keycloak.org/server/features[Enabling and disabling features] {section}.
 The https://www.keycloak.org/high-availability/concepts-memory-and-cpu-sizing[sizing guide] contains a new paragraph describing the updated resource requirements when this feature is enabled.
 
-NOTE: If this feature is enabled for an existing deployment that is using only the embedded Infinispan for storing sessions, the existing online user and client sessions will not be migrated to the database. It will only affect newly created online user and online client sessions.
+For information on how to upgrade, see the link:{upgradingguide_link}[{upgradingguide_name}].
-
-With persistent sessions enabled, the in-memory caches for online user sessions, offline user sessions, online client sessions and offline client sessions are limited to 10000 entries per node by default which will reduce the overall memory usage of Keycloak for larger installations.
-Items which are evicted from memory will be loaded on-demand from the database when needed.
-To set different sizes for the caches, edit {project_name}'s cache config file to set a `+<memory max-count="..."/>+` for those caches.
-Once this feature is enabled, expect an increased database utilization on each login, logout and refresh token request.
-
-To configure the cache size in an external {jdgserver_name} in a {project_name} multi-site setup, for those caches, consult the updated https://www.keycloak.org/high-availability/deploy-infinispan-kubernetes-crossdc[Deploy Infinispan for HA with the Infinispan Operator] {section}.
-
-With this feature enabled, the options `spi-user-sessions-infinispan-offline-session-cache-entry-lifespan-override` and `spi-user-sessions-infinispan-offline-client-session-cache-entry-lifespan-override` are no longer available which were used to override the time offline sessions are kept in memory.
-
-To log out all online users sessions of a realm with the `persistent-user-session` feature enabled, use the following steps as before:
-
-. Login to the Admin Console.
-. Select the menu entry *Sessions*.
-. Select the action *Sign out all active sessions*.
 
 = Cookies updates
 

docs/documentation/upgrading/topics/changes/changes-25_0_0.adoc

@@ -80,6 +80,92 @@ In order to dynamically resolve it from request headers, you need to set the `ho
 
 For more details and more comprehensive scenarios, see https://www.keycloak.org/server/hostname[Configuring the hostname (v2)].
 
+= Persistent user sessions
+
+Previous versions of {project_name} stored only offline user and offline client sessions in the databases.
+The new feature `persistent-user-session` stores online user sessions and online client sessions not only in memory, but also in the database.
+This will allow a user to stay logged in even if all instances of {project_name} are restarted or upgraded.
+
+== Enabling persistent user sessions
+
+The feature is a preview feature and disabled by default. To use it, add the following to your build command:
+
+----
+bin/kc.sh build --features=persistent-user-session ...
+----
+
+For more details see the https://www.keycloak.org/server/features[Enabling and disabling features] {section}.
+The https://www.keycloak.org/high-availability/concepts-memory-and-cpu-sizing[sizing guide] contains a new paragraph describing the updated resource requirements when this feature is enabled.
+
+NOTE: If this feature is enabled for an existing deployment that is using only the embedded Infinispan for storing sessions, the existing online user and client sessions will not be migrated to the database. It will only affect newly created online user and online client sessions.
+
+With persistent sessions enabled, the in-memory caches for online user sessions, offline user sessions, online client sessions and offline client sessions are limited to 10000 entries per node by default which will reduce the overall memory usage of Keycloak for larger installations.
+Items which are evicted from memory will be loaded on-demand from the database when needed.
+To set different sizes for the caches, edit {project_name}'s cache config file to set a `+<memory max-count="..."/>+` for those caches.
+Once this feature is enabled, expect an increased database utilization on each login, logout and refresh token request.
+
+To configure the cache size in an external {jdgserver_name} in a {project_name} multi-site setup, consult the updated https://www.keycloak.org/high-availability/deploy-infinispan-kubernetes-crossdc[Deploy Infinispan for HA with the Infinispan Operator] {section}.
+
+With this feature enabled, the options `spi-user-sessions-infinispan-offline-session-cache-entry-lifespan-override` and `spi-user-sessions-infinispan-offline-client-session-cache-entry-lifespan-override` are no longer available, as they were previously used to override the time offline sessions were kept in-memory.
+
+== Migrating user sessions during the upgrade
+
+When upgrading from {project_name} 24 or earlier, admins can choose to migrate existing online user and client sessions to persistent sessions.
+For this to work, those existing sessions need to be stored in either a remote {jdgserver_name} or in a database configured as JDBC persistence for {project_name}'s embedded cache.
+Migrating in-memory sessions for {project_name} 24 is not supported as all {project_name} instances need to be shut down before the upgrade due to a major version upgrade of the embedded Infinispan.
+
+[WARNING]
+====
+The migration of user sessions only works when the persistent user sessions is enabled when upgrading to {project_name} 25.
+If you chose to upgrade to 25 without enabling persistent user sessions, there is currently no possibility to trigger the migration of existing sessions at a later point in time.
+When you enable it later, only newly created user sessions will be persisted.
+====
+
+To migrate the user sessions during an upgrade of {project_name}, perform the following steps:
+
+. Stop all running old instances of {project_name}.
+
+. Create backups:
++
+--
+* Create a backup {project_name}'s database.
+* If JDBC persistence is used, create a backup of that database if you want to be able to retry the migration of the sessions.
+* If an external {jdgserver_name} is used, create a backup of its data if you want to be able to retry the migration of the sessions.
+--
+. Start the new instances {project_name} with the persistent user sessions feature enabled.
++
+The first starting node will:
++
+--
+. Migrate the database to the schema version 25.
+. Copy all session information from either the remote {jdgserver_name} or the JDBC persistence configured for {project_name}'s embedded cache to the database of {project_name}.
++
+The data will be stored in the tables `offline_user_session` and `online_user_session` with `offline_flag` set to `false`.
+
+. Clear the caches.
++
+This includes clearing the caches of the external {jdgserver_name} if one is used, and clearing the JDBC persistence if one is used.
+--
+. Update the cache configuration XML of {project_name} for caches `sessions` and `clientSessions`:
++
+--
+* If JDBC persistence is used, remove the configuration for JDBC persistence.
+* If the remote {jdgserver_name} has been used in a single-site setup solely for keeping user sessions across {project_name} restarts, remove the remote {jdgserver_name} configuration for those caches.
+--
++
+TIP: If the remote {jdgserver_name} is used in a multi-site setup, you can reduce the resource consumption by the external Infinispan by configuring the number of entries in memory. Use the settings outlined in https://www.keycloak.org/high-availability/deploy-infinispan-kubernetes-crossdc[Deploy Infinispan for HA with the Infinispan Operator] {section}.
+
+. Rolling restart of {project_name} to activate the new cache configuration XML.
+
+== Signing out existing users
+
+In previous versions and when the feature is disabled, a restart of all {project_name} nodes logged out all users.
+To sign out all online users sessions of a realm with the `persistent-user-session` feature enabled, use the following steps as before:
+
+. Login to the Admin Console.
+. Select the menu entry *Sessions*.
+. Select the action *Sign out all active sessions*.
+
 = Metrics for embedded caches enabled by default
 
 Metrics for the embedded caches are now enabled by default.

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/InfinispanUserSessionProviderFactory.java

@@ -211,10 +211,12 @@ public class InfinispanUserSessionProviderFactory implements UserSessionProvider
                 }
             }
         });
-        persistentSessionsWorker = new PersistentSessionsWorker(factory,
+        if (Profile.isFeatureEnabled(Profile.Feature.PERSISTENT_USER_SESSIONS)) {
-                asyncQueuePersistentUpdate,
+            persistentSessionsWorker = new PersistentSessionsWorker(factory,
-                maxBatchSize);
+                    asyncQueuePersistentUpdate,
-        persistentSessionsWorker.start();
+                    maxBatchSize);
+            persistentSessionsWorker.start();
+        }
     }
 
     // Max count of worker errors. Initialization will end with exception when this number is reached
@@ -412,7 +414,9 @@ public class InfinispanUserSessionProviderFactory implements UserSessionProvider
 
     @Override
     public void close() {
-        persistentSessionsWorker.stop();
+        if (persistentSessionsWorker != null) {
+            persistentSessionsWorker.stop();
+        }
     }
 
     @Override

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/PersistentUserSessionProvider.java

@@ -17,11 +17,14 @@
 
 package org.keycloak.models.sessions.infinispan;
 
+import io.reactivex.rxjava3.core.Flowable;
 import org.infinispan.Cache;
 import org.infinispan.client.hotrod.RemoteCache;
 import org.infinispan.client.hotrod.exceptions.HotRodClientException;
 import org.infinispan.commons.api.BasicCache;
 import org.infinispan.context.Flag;
+import org.infinispan.factories.ComponentRegistry;
+import org.infinispan.persistence.manager.PersistenceManager;
 import org.jboss.logging.Logger;
 import org.keycloak.cluster.ClusterProvider;
 import org.keycloak.common.Profile;
@@ -29,6 +32,7 @@ import org.keycloak.common.Profile.Feature;
 import org.keycloak.common.util.Retry;
 import org.keycloak.common.util.Time;
 import org.keycloak.connections.infinispan.InfinispanUtil;
+import org.keycloak.migration.ModelVersion;
 import org.keycloak.models.AuthenticatedClientSessionModel;
 import org.keycloak.models.ClientModel;
 import org.keycloak.models.KeycloakSession;
@@ -42,10 +46,13 @@ import org.keycloak.models.UserSessionProvider;
 import org.keycloak.models.light.LightweightUserAdapter;
 import org.keycloak.models.session.UserSessionPersisterProvider;
 import org.keycloak.models.sessions.infinispan.changes.ClientSessionPersistentChangelogBasedTransaction;
+import org.keycloak.models.sessions.infinispan.changes.JpaChangesPerformer;
+import org.keycloak.models.sessions.infinispan.changes.MergedUpdate;
 import org.keycloak.models.sessions.infinispan.changes.PersistentUpdate;
 import org.keycloak.models.sessions.infinispan.changes.SerializeExecutionsByKey;
 import org.keycloak.models.sessions.infinispan.changes.SessionEntityWrapper;
 import org.keycloak.models.sessions.infinispan.changes.SessionUpdateTask;
+import org.keycloak.models.sessions.infinispan.changes.SessionUpdatesList;
 import org.keycloak.models.sessions.infinispan.changes.Tasks;
 import org.keycloak.models.sessions.infinispan.changes.UserSessionPersistentChangelogBasedTransaction;
 import org.keycloak.models.sessions.infinispan.changes.sessions.CrossDCLastSessionRefreshStore;
@@ -64,10 +71,12 @@ import org.keycloak.models.sessions.infinispan.stream.UserSessionPredicate;
 import org.keycloak.models.sessions.infinispan.util.FuturesHelper;
 import org.keycloak.models.sessions.infinispan.util.InfinispanKeyGenerator;
 import org.keycloak.models.sessions.infinispan.util.SessionTimeouts;
+import org.keycloak.models.utils.KeycloakModelUtils;
 import org.keycloak.models.utils.UserModelDelegate;
 
 import java.nio.charset.StandardCharsets;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
 import java.util.Objects;
@@ -973,4 +982,72 @@ public class PersistentUserSessionProvider implements UserSessionProvider, Sessi
         // This allows creating a UUID that is constant even if the entry is reloaded from the database
         return UUID.nameUUIDFromBytes((userSessionId + clientId).getBytes(StandardCharsets.UTF_8));
     }
+
+    @Override
+    public void migrate(String modelVersion) {
+        if (new ModelVersion(modelVersion).equals(new ModelVersion("25.0.0"))) {
+            migrateNonPersistentSessionsToPersistentSessions();
+        }
+    }
+
+    /**
+     * Copy over all sessions in Infinispan to the persistent user sessions in the database.
+     * This method is public so people can use it to build their custom migrations or re-import sessions when necessary
+     * in a future version of Keycloak.
+     */
+    public void migrateNonPersistentSessionsToPersistentSessions() {
+        JpaChangesPerformer<String, UserSessionEntity> userSessionPerformer = new JpaChangesPerformer<>(sessionCache.getName(), new ArrayBlockingQueue<>(1));
+        JpaChangesPerformer<UUID, AuthenticatedClientSessionEntity> clientSessionPerformer = new JpaChangesPerformer<>(clientSessionCache.getName(), new ArrayBlockingQueue<>(1));
+        AtomicInteger currentBatch = new AtomicInteger(0);
+        var persistence = ComponentRegistry.componentOf(sessionCache, PersistenceManager.class);
+        if (persistence != null && !persistence.getStoresAsString().isEmpty()) {
+            Flowable.fromPublisher(persistence.<String, SessionEntityWrapper<UserSessionEntity>>publishEntries(true, false))
+                    .blockingSubscribe(e -> processEntryFromCache(e.getValue(), userSessionPerformer, clientSessionPerformer, currentBatch));
+        } else {
+            // Usually we assume sessions are stored in a persistence. To be extra safe, iterate over local sessions if no persistent is available.
+            sessionCache.forEach((key, value) -> processEntryFromCache(value, userSessionPerformer, clientSessionPerformer, currentBatch));
+        }
+        flush(userSessionPerformer, clientSessionPerformer);
+        // Clear existing sessions as the IDs of the client sessions have changed.
+        sessionCache.clear();
+        clientSessionCache.clear();
+        // Even though offline sessions haven't been migrated, they are cleared as the IDs of the client sessions have changed. It is safe to clear them as they are already stored in the database.
+        offlineSessionCache.clear();
+        offlineClientSessionCache.clear();
+        log.infof("Migrated %d user sessions total.", currentBatch.intValue());
+    }
+
+    /**
+     * When calling this, ensure that the cache doesn't contain entries for user or client sessions that are already contained in the database.
+     * Such entries should first be cleared from the cache before this is being called.
+     * As this is assumed to run once during the upgrade to Keycloak 25, this should be safe to assume.
+     */
+    private void processEntryFromCache(SessionEntityWrapper<UserSessionEntity> sessionEntityWrapper, JpaChangesPerformer<String, UserSessionEntity> userSessionPerformer, JpaChangesPerformer<UUID, AuthenticatedClientSessionEntity> clientSessionPerformer, AtomicInteger count) {
+        RealmModel realm = session.realms().getRealm(sessionEntityWrapper.getEntity().getRealmId());
+        sessionEntityWrapper.getEntity().getAuthenticatedClientSessions().forEach((k, uuid) -> {
+            SessionEntityWrapper<AuthenticatedClientSessionEntity> clientSession = clientSessionCache.get(uuid);
+            if (clientSession != null) {
+                clientSession.getEntity().setUserSessionId(sessionEntityWrapper.getEntity().getId());
+                MergedUpdate<AuthenticatedClientSessionEntity> merged = MergedUpdate.computeUpdate(Collections.singletonList(Tasks.addIfAbsentSync()), clientSession, 1, 1);
+                clientSessionPerformer.registerChange(Map.entry(uuid, new SessionUpdatesList<>(realm, clientSession)), merged);
+            }
+        });
+        MergedUpdate<UserSessionEntity> merged = MergedUpdate.computeUpdate(Collections.singletonList(Tasks.addIfAbsentSync()), sessionEntityWrapper, 1, 1);
+        userSessionPerformer.registerChange(Map.entry(sessionEntityWrapper.getEntity().getId(), new SessionUpdatesList<>(realm, sessionEntityWrapper)), merged);
+        if (count.incrementAndGet() % 100 == 0) {
+            flush(userSessionPerformer, clientSessionPerformer);
+        }
+        if (count.intValue() % 1000 == 0) {
+            log.infof("Migrated %d user sessions total, continuing...", count.intValue());
+        }
+    }
+
+    private <E extends SessionEntity, K> void flush(JpaChangesPerformer<K, E> userSessionsPerformer, JpaChangesPerformer<UUID, AuthenticatedClientSessionEntity> clientSessionPerformer) {
+        KeycloakModelUtils.runJobInTransaction(session.getKeycloakSessionFactory(),
+                s -> {
+                    userSessionsPerformer.applyChangesSynchronously(s);
+                    clientSessionPerformer.applyChangesSynchronously(s);
+                });
+    }
+
 }

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/changes/JpaChangesPerformer.java

@@ -113,6 +113,14 @@ public class JpaChangesPerformer<K, V extends SessionEntity> implements SessionC
                 exceptions.forEach(ex::addSuppressed);
                 throw ex;
             }
+            changes.clear();
+        }
+    }
+
+    public void applyChangesSynchronously(KeycloakSession session) {
+        if (!changes.isEmpty()) {
+            changes.forEach(persistentUpdate -> persistentUpdate.perform(session));
+            changes.clear();
         }
     }
 

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/changes/SessionUpdatesList.java

@@ -30,7 +30,7 @@ import org.keycloak.models.sessions.infinispan.entities.SessionEntity;
  *
  * @author <a href="mailto:mposolda@redhat.com">Marek Posolda</a>
  */
-class SessionUpdatesList<S extends SessionEntity> {
+public class SessionUpdatesList<S extends SessionEntity> {
 
     private final RealmModel realm;
 

model/storage-private/src/main/java/org/keycloak/migration/migrators/MigrateTo25_0_0.java

@@ -21,11 +21,13 @@ package org.keycloak.migration.migrators;
 
 
 import org.jboss.logging.Logger;
+import org.keycloak.common.Profile;
 import org.keycloak.migration.MigrationProvider;
 import org.keycloak.migration.ModelVersion;
 import org.keycloak.models.ClientScopeModel;
 import org.keycloak.models.KeycloakSession;
 import org.keycloak.models.RealmModel;
+import org.keycloak.models.utils.KeycloakModelUtils;
 import org.keycloak.representations.idm.RealmRepresentation;
 
 /**
@@ -59,6 +61,9 @@ public class MigrateTo25_0_0 implements Migration {
 
         //add basic scope to existing clients
         realm.getClientsStream().forEach(c-> c.addClientScope(basicScope, true));
+
+        // offer a migration for persistent user sessions which was added in KC25
+        session.sessions().migrate(VERSION.toString());
     }
 }
 

server-spi/src/main/java/org/keycloak/models/UserSessionProvider.java

@@ -17,6 +17,7 @@
 
 package org.keycloak.models;
 
+import org.keycloak.migration.MigrationModel;
 import org.keycloak.provider.Provider;
 
 import java.util.Collection;
@@ -208,4 +209,7 @@ public interface UserSessionProvider extends Provider {
     void close();
 
     int getStartupTime(RealmModel realm);
+
+    default void migrate(String modelVersion) {
+    }
 }

testsuite/model/pom.xml

@@ -219,6 +219,13 @@
             </properties>
         </profile>
 
+        <profile>
+            <id>jpa+cross-dc-infinispan+persistentsessions</id>
+            <properties>
+                <keycloak.model.parameters>CrossDCInfinispan,Jpa,PersistentUserSessions</keycloak.model.parameters>
+            </properties>
+        </profile>
+
         <profile>
             <id>jpa+infinispan+client-storage</id>
             <properties>

testsuite/model/src/test/java/org/keycloak/testsuite/model/session/UserSessionPersisterProviderTest.java

@@ -18,6 +18,7 @@
 package org.keycloak.testsuite.model.session;
 
 import org.junit.Assert;
+import org.junit.Assume;
 import org.junit.Test;
 import org.keycloak.OAuth2Constants;
 import org.keycloak.common.Profile;
@@ -32,7 +33,9 @@ import org.keycloak.models.UserModel;
 import org.keycloak.models.UserProvider;
 import org.keycloak.models.UserSessionModel;
 import org.keycloak.models.UserSessionProvider;
+import org.keycloak.models.jpa.session.JpaUserSessionPersisterProvider;
 import org.keycloak.models.session.UserSessionPersisterProvider;
+import org.keycloak.models.sessions.infinispan.PersistentUserSessionProvider;
 import org.keycloak.models.utils.ResetTimeOffsetEvent;
 import org.keycloak.protocol.oidc.OIDCLoginProtocol;
 import org.keycloak.protocol.oidc.OIDCLoginProtocolFactory;
@@ -557,6 +560,43 @@ public class UserSessionPersisterProviderTest extends KeycloakModelTest {
         }
     }
 
+    @Test
+    public void testMigrateSession() {
+        Assume.assumeTrue(Profile.isFeatureEnabled(Profile.Feature.PERSISTENT_USER_SESSIONS));
+
+        UserSessionModel[] sessions = inComittedTransaction(session -> {
+            // Create some sessions in infinispan
+            return createSessions(session, realmId);
+        });
+
+        inComittedTransaction(session -> {
+            // clear the entries in the database to enable the migration
+            JpaUserSessionPersisterProvider sessionPersisterProvider = (JpaUserSessionPersisterProvider) session.getProvider(UserSessionPersisterProvider.class);
+            sessionPersisterProvider.removeUserSessions(session.realms().getRealm(realmId), false);
+
+            // verify that clearing was successful
+            Assert.assertEquals(0, countUserSessionsInRealm(session));
+        });
+
+        inComittedTransaction(session -> {
+            // trigger a migration with the entries that are still in the cache
+            PersistentUserSessionProvider userSessionProvider = (PersistentUserSessionProvider) session.getProvider(UserSessionProvider.class);
+            userSessionProvider.migrateNonPersistentSessionsToPersistentSessions();
+            JpaUserSessionPersisterProvider sessionPersisterProvider = (JpaUserSessionPersisterProvider) session.getProvider(UserSessionPersisterProvider.class);
+
+            // verify that import was complete
+            Assert.assertEquals(sessions.length, countUserSessionsInRealm(session));
+        });
+    }
+
+    private long countUserSessionsInRealm(KeycloakSession session) {
+        JpaUserSessionPersisterProvider sessionPersisterProvider = (JpaUserSessionPersisterProvider) session.getProvider(UserSessionPersisterProvider.class);
+        RealmModel realm = session.realms().getRealm(realmId);
+        return sessionPersisterProvider.getUserSessionsCountsByClients(realm, false).keySet().stream()
+                .flatMap(s -> sessionPersisterProvider.loadUserSessionsStream(realm, session.clients().getClientById(realm, s), false, 0, -1))
+                .distinct().count();
+    }
+
     private void setupClientStorageComponents(KeycloakSession s, RealmModel realm) {
         getParameters(ClientStorageProviderModel.class).forEach(cm -> {
             cm.put(HardcodedClientStorageProviderFactory.CLIENT_ID, "external-storage-client");