Merge pull request #31 from patriot1burke/master

KEYCLOAK-4527 KEYCLOAK-4404

authorization_services/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)
 

getting_started/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)
 

securing_apps/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)
 

server_admin/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)
 

server_admin/topics/identity-broker/configuration.adoc

@@ -52,6 +52,13 @@ Regardless the identity provider you are creating, you'll see the following conf
 |Enabled
 |Turn the provider on/off
 
+|Hide On Login Page
+|When this switch is on, this provider will not be shown as a login option on the login page.  Clients can still request to use this provider by using the 'kc_idp_hint' parameter in the URL they use to request a login.
+
+|Link Only
+|When this switch is on, this provider cannot be used to login users and will not be shown as an option on the login page.  Existing accounts can still be linked with this provider though.
+
+
 |Store Tokens
 |Whether or not to store the token received from the identity provider.
 

server_admin/topics/identity-broker/tokens.adoc

@@ -19,6 +19,8 @@ An application must have authenticated with {{book.project.name}} and have recei
 will need to have the `broker` client-level role `read-token` set.  This means that the user must have a role mapping for this role
 and the client application must have that role within its scope.
 In this case, given that you are accessing a protected service in {{book.project.name}}, you need to send the access token issued by {{book.project.name}} during the user authentication.
-
 In the broker configuration page you can automatically assign this role to newly imported users by turning on the `Stored Tokens Readable` switch.
 
+These external tokens can be re-established by either logging in again through the provider, or using the client initiated account linking API.
+
+

server_admin/topics/user-federation.adoc

@@ -2,23 +2,21 @@
 
 == User Storage Federation
 
-{{book.project.name}} can federate external user databases.
-Out of the box we have support for LDAP and Active Directory.
-Before you dive into this, you should understand how {{book.project.name}} does federation.
+Many companies have existing user databases that hold information about users and their passwords or other credentials.
+In may cases, it is just not possible to migrate off of those existing stores to a pure {{book.project.name}} deployment.
+{{book.project.name}} can federate existing external user databases.
+Out of the box we have support for LDAP and Active Directory.  You can also code your own extension for any custom
+user databases you might have using our User Storage SPI.
 
-{{book.project.name}} performs federation a bit differently than other products/projects.
-The vision of {{book.project.name}} is that it is an out of the box solution that should provide a core set of features regardless of the backend user storage you want to use.
-Because of this requirement/vision, {{book.project.name}} has a set data model that all of its services use.
-Most of the time when you want to federate an external user store, much of the metadata that would be needed to provide this complete feature set does not exist in that external store.
-For example your LDAP server may only provide password validation, but not support TOTP or user role mappings.
-The {{book.project.name}} User Federation SPI was written to support these completely variable configurations.
+The way it works is that when a user logs in, {{book.project.name}} will look into its own internal user store to find the user.
+If it can't find it there it will iterate
+over every User Storage provider you have configured for the realm until it finds a match.  Data from the external store is mapped into a common user model that is consumed by the {{book.project.name}}
+runtime.  This common user model can then be mapped to OIDC token claims and SAML assertion attributes.
 
-The way user federation works is that {{book.project.name}} will import your federated users on demand to its local storage.
-How much metadata is imported depends on the underlying federation plugin and how that plugin is configured.
-Some federation plugins may only import the username into {{book.project.name}} storage. Others might import everything from name, address, and phone number, to user role mappings.
-Some plugins might want to import credentials directly into {{book.project.name}} storage and let {{book.project.name}} handle credential validation.
-Others might want to handle credential validation themselves.
-The goal of the User Storage Federation SPI is to support all of these scenarios.
+External user databases rarely have every piece of data need to support all the features that {{book.project.name}} has.
+In this case, the User Storage Provider can opt to store some things locally in the {{book.project.name}} user store.
+Some providers even import the user locally and sync periodically with the external store.  All this depends on the capabilities of the provider and how its configured.  For example, your
+external user store may not support OTP.  Depending on the provider, this OTP support can be handled and stored by {{book.project.name}}
 
 === Adding a Provider
 
@@ -27,5 +25,5 @@ To add a storage provider go to the `User Federation` left menu item in the Admi
 .User Federation
 image:../{{book.images}}/user-federation.png[]
 
-On the right side, there is an `Add Provider` list box.  Choose the provider you want to add and you will be brought to the configuration page of that provider.
+On the right side, there is an `Add Provider` list box.  Choose the provider type you want to add and you will be brought to the configuration page of that provider.
 

server_admin/topics/user-federation/custom.adoc

@@ -1,6 +1,5 @@
 
 === Custom Providers
 
-{{book.project.name}} does have a private SPI for User Storage Federation that you can use to write your own custom providers.
-There is no commercial support for this yet.  You can find some documentation for this in our link:https://keycloak.gitbooks.io/server-developer-guide/content/[community documentation].
-This SPI is slated for a major rewrite before commercial support will be provided.
+{{book.project.name}} does have an SPI for User Storage Federation that you can use to write your own custom providers.
+You can find documentation for this in our link:{{book.developerguide.link}}[{{book.developerguide.name}}].

server_admin/topics/user-federation/ldap.adoc

@@ -2,16 +2,36 @@
 
 === LDAP and Active Directory
 
-{{book.project.name}} comes with a built-in LDAP/AD plugin.
-By default, it is set up only to import username, email, first name, and last name. But you are free to configure additional <<_ldap_mappers,mappers>>
-and add more attributes or delete the default ones.
-It supports password validation via LDAP/AD protocols and different user metadata synchronization modes.
+{{book.project.name}} comes with a built-in LDAP/AD provider.  It is possible to federate multiple different LDAP servers in the same
+{{book.project.name} realm.  You can map LDAP user attributes into the {{book.project.name}} common user model.
+By default, it maps username, email, first name, and last name, but you are free to configure additional <<_ldap_mappers,mappings>>.
+The LDAP provider also supports password validation via LDAP/AD protocols and different storage, edit, and synchronization modes.
+
 To configure a federated LDAP store go to the Admin Console.
 Click on the `User Federation` left menu option.
 When you get to this page there is an `Add Provider` select box.
 You should see _ldap_ within this list.
 Selecting _ldap_ will bring you to the ldap configuration page.
 
+==== Storage Mode
+
+By default, {{book.project.name}} will import users from LDAP into the local {{book.project.name}} user database. This copy of the user
+is either synchronized on demand, or through a periodic background task.
+The one exception to this is passwords.  Passwords are not imported and password validation is
+delegated to the LDAP server.  The benefits to this approach is that all {{book.project.name}} features will work as any extra
+per-user data that is needed can be stored locally.  This approach also reduces load on the LDAP server as uncached users are loaded
+from the {{book.project.name}} database the 2nd time they are accessed.  The only load your LDAP server will have is password validation.
+The downside to this approach is that when a user is first queried, this will require a {{book.project.name}} database insert.  The import will
+also have to be synchronized with your LDAP server as needed.
+
+Alternatively, you can choose not to import users into the {{book.project.name}} user database.  In this case, the common user model
+that the  {{book.project.name}} runtime uses is backed only by the LDAP server.  This means that if LDAP doesn't support
+a piece of data that a {{book.project.name}} feature needs that feature will not work.
+The benefit to this approach is that you do not have the overhead of importing and synchronizing a copy of the LDAP user into the
+{{book.project.name}} user database.
+
+This storage mode is controled by the `Import Enabled` switch.  Set to `On` to import users.
+
 ==== Edit Mode
 
 Users, through the <<fake/../../account.adoc#_account-service, User Account Service>>, and admins through the Admin Console
@@ -29,7 +49,7 @@ WRITABLE::
 UNSYNCED::
   Any changes to username, email, first name, last name, and passwords will be stored in {{book.project.name}} local storage.
   It is up to you to figure out how to synchronize back to LDAP. This allows {{book.project.name}} deployments to support
-  updates of user metadata on a read-only LDAP server.
+  updates of user metadata on a read-only LDAP server.  This option only applies when you are importing users from LDAP into the local {{book.project.name}} user database.
 
 ==== Other config options
 
@@ -40,7 +60,8 @@ Priority::
   The priority of this provider when looking up users or for adding registrations.
 
 Sync Registrations::
-  If a new user is added through a registration page or admin console, should the user be eligible to be synchronized to this provider?
+  Does your LDAP support adding new users?  Click this switch if you want new users created by {{book.project.name}} in the admin console or the registration page
+  to be added to LDAP.
 
 Allow Kerberos authentication::
   Enable Kerberos/SPNEGO authentication in realm with users data provisioned from LDAP.
@@ -66,8 +87,8 @@ if the connection to LDAP starts with `ldaps`.
 
 ==== Sync of LDAP users to {{book.project.name}}
 
-LDAP Federation Provider will automatically take care of synchronization (import) of needed LDAP users into the {{book.project.name}} local database.
-As users log in, the LDAP Federation provider will import the LDAP user
+If you have import enabled, the LDAP Provider will automatically take care of synchronization (import) of needed LDAP users into the {{book.project.name}} local database.
+As users log in, the LDAP provider will import the LDAP user
 into the {{book.project.name}} database and then authenticate against the LDAP password. This is the only time users will be imported.
 If you go to the `Users` left menu item in the Admin Console and click the `View all users` button, you will only see those LDAP users that
 have been authenticated at least once by {{book.project.name}}.  It is implemented this way so that admins don't accidentally try to import a huge LDAP DB of users.
@@ -87,9 +108,10 @@ The best way to handle syncing is to click the `Synchronize all users` button wh
 then set up a periodic sync of changed users.  The configuration page for your LDAP Provider has several options to support you.
 
 [[_ldap_mappers]]
-==== LDAP/Federation mappers
 
-LDAP mappers are `listeners`, which are triggered by the LDAP Federation provider at various points, provide another extension point to LDAP integration.
+==== LDAP Mappers
+
+LDAP mappers are `listeners`, which are triggered by the LDAP Provider at various points, provide another extension point to LDAP integration.
 They are triggered when a user logs in via LDAP and needs to be imported, during {{book.project.name}} initiated registration, or when a user is queried from the Admin Console.
 When you create an LDAP Federation provider, {{book.project.name}} will automatically provide set of built-in `mappers` for this provider.
 You are free to change this set and create a new mapper or update/delete existing ones.

server_development/SUMMARY.adoc

@@ -4,6 +4,9 @@
  .. link:server_development/topics/admin-rest-api.adoc[Admin REST API]
  .. link:server_development/topics/themes.adoc[Themes]
  .. link:server_development/topics/custom-attributes.adoc[Custom User Attributes]
+ .. link:server_development/topics/identity-brokering.adoc[Identity Brokering APIs]
+ ... link:server_development/topics/identity-brokering/tokens.adoc[Retrieving External IDP Tokens]
+ ... link:server_development/topics/identity-brokering/account-linking.adoc[Client Initiated Account Linking]
 {% if book.community %}
  .. link:server_development/topics/providers.adoc[Service Provider Interfaces (SPI)]
  .. link:server_development/topics/extensions.adoc[Extending Server]

server_development/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)
 

server_development/topics/identity-brokering.adoc

@@ -0,0 +1,8 @@
+
+== Identity Brokering APIs
+
+{{book.project.name}} can delegate authentication to a parent IDP for login.  A typical example of this is the case
+where you want users to be able to login through a social provider like Facebook or Google.  {{book.project.name}}
+also allows you to link existing accounts to a brokered IDP.  This section talks about some APIs that your applications
+can use as it pertains to identity brokering.
+

server_development/topics/identity-brokering/account-linking.adoc

@@ -0,0 +1,94 @@
+
+=== Client Initiated Account Linking
+
+Some applications want to integrate with social providers like Facebook, but do not want to provide an option to login via
+these social providers.  {{book.project.name}} offers a browser-based API that applications can use to link an existing
+user account to a specific external IDP.  This is called client initiated account linking.
+
+The way it works is that the application forward's the user's browser to a URL on the {{book.project.name}} server requesting
+that it wants to link the user's account to a specific external provider (i.e. Facebook).  The server
+initiates a login with the external provider.  The browser logs in at the external provider and is redirected
+back to the auth server.  The auth server establishes the link and redirects back to the application with a confirmation.
+
+There are some preconditions that must be met by the client application before it can initiate this protocol:
+
+* The desired identity provider must be configured and enabled for the user's realm in the admin console.
+* The application must already be logged in as an existing user via the OIDC protocol
+* The user must have an `account.manage-account` or `account.manage-account-links` role mapping.
+* The application must be granted the scope for those roles within its access token
+* The application must have access to its access token as it needs information within it to generate the redirect URL.
+
+To initiate the login, the application must fabricate a URL and redirect the user's browser to this URL.  The URL looks like this:
+
+[source,java]
+----
+{auth-server-root}/auth/realms/{realm}/broker/{provider}/linking?client_id={id}&redirect_uri={uri}&nonce={nonce}&hash={hash}
+----
+
+Here's a description of each path and query param:
+
+provider::
+  This is the provider alias of the external IDP that you defined in the `Identity Provider` section of the admin console.
+
+client_id::
+  This is the OIDC client id of your application.  When you registered the application as a client in the admin console,
+  you had to specify this client id.
+
+redirect_uri::
+  This is the application callback URL you want to redirect to after the account link is established.  It must be a valid
+  client redirect URI pattern.  In other words, it must match one of the valid URL patterns you defined when you registered
+  the client in the admin console.
+
+nonce::
+  This is a random string that your application must generate
+
+hash::
+  This is a Base64 URL encoded hash.  This hash is generated by Base64 URL encoding a SHA_256 hash of `nonce` + `token.getSessionState()` + `token.getClientSession()` + `provider`
+  The token variable are obtained from the OIDC access token.  Basically you are hashing the random nonce, the user session id, the client session id, and the identity
+  provider alias you want to access.
+
+Here's an example of Java Servlet code that generates the URL to establish the account link.
+
+
+[source,java]
+----
+   KeycloakSecurityContext session = (KeycloakSecurityContext) httpServletRequest.getAttribute(KeycloakSecurityContext.class.getName());
+   AccessToken token = session.getToken();
+   String clientSessionId = token.getClientSession();
+   String nonce = UUID.randomUUID().toString();
+   MessageDigest md = null;
+   try {
+      md = MessageDigest.getInstance("SHA-256");
+   } catch (NoSuchAlgorithmException e) {
+      throw new RuntimeException(e);
+   }
+   String input = nonce + token.getSessionState() + clientSessionId + provider;
+   byte[] check = md.digest(input.getBytes(StandardCharsets.UTF_8));
+   String hash = Base64Url.encode(check);
+   request.getSession().setAttribute("hash", hash);
+   String redirectUri = ...;
+   String accountLinkUrl = KeycloakUriBuilder.fromUri(authServerRootUrl)
+                    .path("/auth/realms/{realm}/broker/{provider}/link")
+                    .queryParam("nonce", nonce)
+                    .queryParam("hash", hash)
+                    .queryParam("client_id", token.getIssuedFor())
+                    .queryParam("redirect_uri", redirectUri).build(realm, provider).toString();
+----
+
+Why is this hash included?  We do this so that the auth server is guaranteed to know that the client application initiated the request and no other rogue app
+just randomly asked for a user account to be linked to a specific provider.  The auth server will first check to see if the user is logged in by checking the SSO
+cookie set at login.  It will then try to regenerate the hash based on the current login and match it up to the hash sent by the application.
+
+After the account has been linked, the auth server will redirect back to the `redirect_uri`.  If there is a problem servicing the link request,
+the auth server may or may not redirect back to the `redirect_uri`.  The browser may just end up at an error page instead of being redirected back
+to the application.  If there is an error condition and the auth server deems it safe enough to redirect back to the client app, an additional
+`error` query parameter will be appended to the `redirect_uri`.
+
+[WARNING]
+   While this API guarantees that the application initiated the request, it does not completely prevent CSRF attacks for this operation.  The application
+   is still responsible for guarding against CSRF attacks target at itself.
+
+==== Refreshing External Tokens
+
+If you are using the external token generated by logging into the provider (i.e. a Facebook or Github token), you can refresh this token by re-initiating the account linking API.
+

server_development/topics/identity-brokering/tokens.adoc

@@ -0,0 +1,26 @@
+
+=== Retrieving External IDP Tokens
+
+{{book.project.name}} allows you to store tokens and responses from the authentication process with the external IDP.
+For that, you can use the `Store Token` configuration option on the IDP's settings page.
+
+Application code can retrieve these tokens and responses to pull in extra user information, or to securely invoke requests on the external IDP.
+For example, an application might want to use the Google token to invoke on other Google services and REST APIs.
+To retrieve a token for a particular identity provider you need to send a request as follows:
+
+[source,java]
+----
+GET /auth/realms/{realm}/broker/{provider_alias}/token HTTP/1.1
+Host: localhost:8080
+Authorization: Bearer {keycloak_access_token}
+----
+
+An application must have authenticated with {{book.project.name}} and have received an access token.  This access token
+will need to have the `broker` client-level role `read-token` set.  This means that the user must have a role mapping for this role
+and the client application must have that role within its scope.
+In this case, given that you are accessing a protected service in {{book.project.name}}, you need to send the access token issued by {{book.project.name}} during the user authentication.
+In the broker configuration page you can automatically assign this role to newly imported users by turning on the `Stored Tokens Readable` switch.
+
+These external tokens can be re-established by either logging in again through the provider, or using the client initiated account linking API.
+
+

server_installation/gitlab-conversion.py

@@ -66,7 +66,7 @@ output.write("""
 include::document-attributes.adoc[]
 """)
 
-input = re.sub(r"[ ]*\.+\s*link:(.*)\[(.*)\]", "include::\g<1>[]", input)
+input = re.sub(r"[ ]*\.+\s*link:[^/]+/(.*)\[(.*)\]", "include::\g<1>[]", input)
 input = applyTransformation(input)
 output.write(input)