Link SAML logins to accounts by a persistent identifier

docs/conf_sso_test_msscolab.rst

@@ -8,7 +8,10 @@ Testing IDP (`mslib/msidp`) is specifically designed for testing the Single Sign
 Here is documentation that explains the configuration of the MSS Colab Server with the testing IdP.
 
 .. warning::
-   When running publicly rather than in development, you should not use the built-in development server ( msidp / idp.py ).
+   msidp is for development and tests only. Never run it where others can reach it, and never let an MSColab
+   server that others use trust it: its users and passwords are the ones in ``mslib/msidp/idp_user.py``, which
+   everyone knows, and it lacks protections of a real identity provider. For example it accepts unsigned requests,
+   has no CSRF protection on its login form and redirects to any ``redirect_uri``. It prints this warning on start.
 
    The development server is provided by MSS for convenience, but is not designed to be particularly efficient, stable, or secure.
 
@@ -127,5 +130,7 @@ When migrations finished, you can start mscolab server  using the following comm
 
     $ msui
 
-* Login with identity provider through Qt Client application.
+* Login with identity provider through Qt Client application. After the login at msidp the browser shows a
+  one-time code, enter it in msui. If an MSColab account with the email of the msidp user exists already, msui asks
+  for its password once to link the msidp login to it.
 * To log in to the mscolab server through the identity provider, you can use the credentials specified in the ``USERS`` and ``PASSWD`` section of the ``MSS/mslib/msidp/idp_user.py`` file. Look for the relevant section in the file to find the necessary login credentials.

docs/samples/config/mscolab/mss_saml2_backend.yaml.sample

@@ -52,7 +52,8 @@ config:
         want_response_signed: true
         want_assertion_signed: true
         allow_unknown_attributes: true
-        allow_unsolicited: true
+        # only responses to a login request of this server
+        allow_unsolicited: false
         endpoints:
           assertion_consumer_service:
             - [http://localhost:8083/localhost_test_idp/acs/post, 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST']
@@ -106,7 +107,7 @@ config:
   #       want_response_signed: true
   #       want_assertion_signed: true
   #       allow_unknown_attributes: true
-  #       allow_unsolicited: true
+  #       allow_unsolicited: false
   #       endpoints:
   #         assertion_consumer_service:
   #           - [http://localhost:8083/idp2/acs/post, 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST']

docs/samples/config/mscolab/setup_saml2_backend.py.sample

@@ -17,6 +17,9 @@ class setup_saml2_backend:
             'idp_identity_name': 'localhost_test_idp',  #  make sure to use underscore for the blanks
             'idp_data': {
                 'idp_name': 'Testing Identity Provider',  # this name is used on the Login page to connect to the Provider.
+                # the attribute that identifies a user: it must not change and must never be given to someone else.
+                # Without it the IdP must send a persistent NameID.
+                'user_id_attribute': 'eduPersonPrincipalName',
             }
         },
 

docs/sso_via_saml_mscolab.rst

@@ -102,6 +102,7 @@ In this implementation, as we are enabling only one IdP, there is no need to con
                       'idp_identity_name': 'localhost_test_idp',  #  make sure to use underscore for the blanks
                       'idp_data': {
                           'idp_name': 'Testing Identity Provider',  #  this name is used on the Login page to connect to the Provider
+                          'user_id_attribute': 'eduPersonPrincipalName',  #  the attribute that identifies a user
                       }
                   },
           ]
@@ -116,6 +117,12 @@ In this implementation, as we are enabling only one IdP, there is no need to con
 
 	Idp_name refers to the name of the Identity Provider that will be displayed in the MSColab server web interface for end users to select when configuring SSO.
 
+    User_id_attribute is the SAML attribute that identifies a user of this IdP, e.g. ``eduPersonPrincipalName``
+    or ``urn:oasis:names:tc:SAML:attribute:subject-id``. Its value must not change and must never be given to
+    someone else. Without ``user_id_attribute`` the IdP must send a persistent NameID
+    (``urn:oasis:names:tc:SAML:2.0:nameid-format:persistent``). MSColab refuses a login without such an
+    identifier. The email address is no identifier: it can change, and an IdP can assert any address.
+
 
 Establish pysaml2, Saml2Client for the MSColab server
 #####################################################
@@ -189,7 +196,7 @@ Please refer the yaml file template (`mss_saml2_backend.yaml.sample`) in the dir
          want_response_signed: true
          want_assertion_signed: true
          allow_unknown_attributes: true
-         allow_unsolicited: true
+         allow_unsolicited: false
          endpoints:
            assertion_consumer_service:
              - [http://localhost:8083/idp2/acs/post, 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST']
@@ -221,7 +228,48 @@ While the core purpose of IdPs is to authenticate users and provide information
 Guide to IDP Configuration
 --------------------------
 
-In the SSO process through the MSColab server, the username is obtained as ``givenName``, and the email address is obtained as ``email``. Therefore, when configuring the IdP, it is necessary to configure it accordingly to ensure the correct return of the givenName attribute and the email address along with the SAML response.
+In the SSO process through the MSColab server, the username is obtained as ``givenName`` (or ``displayName``), and the email address is obtained as ``email`` (or ``mail``). Therefore, when configuring the IdP, it is necessary to configure it accordingly to ensure the correct return of the givenName attribute and the email address along with the SAML response. The IdP must also send the attribute of ``user_id_attribute`` or a persistent NameID, see above.
+
+How a SAML login finds its account
+----------------------------------
+
+MSColab links an account to SAML identities: the entity id of the IdP together with the value of
+``user_id_attribute`` (or the persistent NameID). A login by a linked identity logs in to its account, also when the
+IdP sends another email address than before.
+
+* The first login by an identity that isn't linked yet gets a new account with the email address and the given name
+  that the IdP sends. The given name becomes the username; if it is taken, contains ``@`` or is longer than 255
+  characters, the part of the email address before the ``@`` does. The browser shows why a login is refused: no valid
+  email address, or both usernames can't be used, or several accounts have the email address in different upper and
+  lower case.
+* If an account with that email address exists already, in any upper and lower case, the login is refused until its
+  owner links the identity: msui asks for the password of that account once and links the identity right after the
+  password login.
+* Accounts of MSColab before 12.0.0 that logged in by SAML have no linked identity yet. Such an account is linked on
+  its next login by email, if it logged in by the same configured IdP before and the response comes from the IdP that
+  MSColab sent the login request to. A response of another IdP of the same metadata, e.g. of a federation, needs the
+  link by the owner, like any other existing account.
+* With ``MAIL_ENABLED`` an account whose email address isn't confirmed can't log in by SAML either; msui tells so, and
+  the code works again once the address is confirmed.
+* A SAML login never changes the password of an account. Accounts created by a SAML login have a random password; their
+  owners can set one by a password reset. A password reset unlinks the SAML identities of the account: whoever resets
+  the password owns the email address. The next SAML login links the identity again, with the new password.
+* Without ``user_id_attribute`` an IdP must send a persistent NameID. The server warns at its start if neither is set
+  up for a configured IdP: every SAML login by it would be refused.
+
+The login in msui
+-----------------
+
+After the login at the IdP the browser shows a one-time code. The user enters it in msui, which exchanges it for a
+login token. The code works once and for 20 minutes; the IdP has to answer the login request within 10 minutes. The
+server accepts only responses to login requests it sent, each once, so ``allow_unsolicited`` must be ``false``; a
+login started at the IdP (IdP-initiated) doesn't work. The requests are stored in the database, so every process of
+the server accepts the response. Anyone can start a login, so the server keeps at most 1000 unfinished ones; further
+logins are refused with "503 Service Unavailable" until some of them expire. With ``ENABLE_BASIC_HTTP_AUTHENTICATION``
+the browser needs the basic authentication of the server to start a login, and msui to exchange the code, as for a
+password login. An assertion that isn't signed counts only if the signed response around it comes from the same IdP.
+msui 12.0.0 also logs in by SAML to servers before 12.0.0; msui before 12.0.0 can't log in by SAML to a 12.0.0
+server.
 
 
 3. Configuration example through Keycloak 13.0.1
@@ -469,6 +517,10 @@ Configuration in MSColab settings for Keycloak
         .. note::
             Make sure to insert idp_identity_name as above ('key_cloak_v_13'), which used in this example.
 
+            This example has no ``user_id_attribute``: set the Name ID Format of the client in Keycloak to
+            ``persistent`` and ``name_id_format`` in `mss_saml2_backend.yaml` to
+            ``urn:oasis:names:tc:SAML:2.0:nameid-format:persistent``, then the NameID identifies the user.
+
 Configuration mss_saml2_backend.yaml file
 
     Create your mss_saml2_backend.yaml file in your ``SSO_DIR``.
@@ -528,7 +580,7 @@ Configuration mss_saml2_backend.yaml file
                     want_response_signed: true
                     want_assertion_signed: true
                     allow_unknown_attributes: true
-                    allow_unsolicited: true
+                    allow_unsolicited: false
                     endpoints:
                       assertion_consumer_service:
                         - [http://localhost:8083/keycloak_idp/acs/post, 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST']

mslib/mscolab/CLAUDE.md

@@ -70,6 +70,13 @@ and `servers-without-qt` contracts in setup.cfg.
   `logout_everywhere`) sets a new one and invalidates all earlier tokens. Email-link tokens use
   `generate_confirmation_token(email, purpose)` with their own salt per purpose, and
   links in emails are built with `public_url_for` from `PUBLIC_URL`, never `_external=True`.
+- A SAML login is accepted by the linked `IdpIdentity` (IdP entity id + configured `user_id_attribute` or
+  persistent NameID), never by email alone, and never changes a password. Only responses to a request in
+  `saml_logins` are accepted, once, at most `MAX_SAML_LOGINS` open at a time; msui exchanges the one-time code at
+  `idp_login_auth` (see `auth.idp_login`). The issuer is the one of the signed assertion (of an unsigned one only if
+  the signed response has the same issuer). A pre-12 account is linked by email only for the IdP the request went
+  to. The code is used up in the same transaction that creates or links the account, so a failure leaves it valid.
+  The ACS routes are added to the app in `init_saml`: a blueprint can't change after its first registration.
 - No module level state that belongs to one app (app, db engine, SocketIO,
   HTTPBasicAuth, managers). It lives on the app: `app.config` or
   `app.extensions['sockio' | 'cm' | 'fm' | 'basic_auth']`, read via `current_app`.

mslib/mscolab/api/endpoints.py

@@ -57,5 +57,6 @@
 STATUS = "status"
 REGISTER = "register"
 IDP_LOGIN_AUTH = "idp_login_auth"
+IDP_LINK = "idp_link"
 MESSAGES = "messages"
 MESSAGE_ATTACHMENT = "message_attachment"

mslib/mscolab/api/schemas.py

@@ -469,9 +469,7 @@ def from_text(cls, text):
 
 @dataclass
 class LoginRequest:
-    """POST endpoints.TOKEN -- log in (email+password), or complete an IDP
-    login by posting the IDP-issued token as the "password".
-    """
+    """POST endpoints.TOKEN -- log in (email+password)."""
 
     email: str
     password: str
@@ -1096,14 +1094,15 @@ def from_text(cls, text):
 
 @dataclass
 class IdpLoginAuthRequest:
-    """POST endpoints.IDP_LOGIN_AUTH -- submit the SAML-issued token to
-    complete an IDP login.
+    """POST endpoints.IDP_LOGIN_AUTH -- exchange the one-time code that the
+    browser shows after a SAML login for a login token.
 
     Unlike every other route migrated so far, the client sends this as a
     real JSON body (`requests.post(url, json=...)`), not form-encoded data,
     and the server reads it via `request.get_json()`, not `request.form` --
     so this is `to_json_data()`/`from_json_data()`, not
-    `to_form_data()`/`from_form()`.
+    `to_form_data()`/`from_form()`. The field is called token for
+    compatibility, it holds the code.
     """
 
     token: str
@@ -1118,21 +1117,22 @@ def from_json_data(cls, data):
 
 @dataclass
 class IdpUserInfo:
-    """User info embedded in IdpLoginAuthResponse -- a different shape than
-    UserInfo (username/id/emailid here, vs. username/id/fullname there; no
-    route shares this exact shape with any other migrated so far).
+    """User info embedded in IdpLoginAuthResponse -- UserInfo plus emailid,
+    so that the response is also a LoginResponse for msui's after_login.
     """
 
     username: str
     id: int  # noqa: A003
     emailid: str
+    fullname: str = ""
 
     def to_dict(self):
-        return {"username": self.username, "id": self.id, "emailid": self.emailid}
+        return {"username": self.username, "id": self.id, "emailid": self.emailid, "fullname": self.fullname}
 
     @classmethod
     def from_dict(cls, data):
-        return cls(username=data["username"], id=data["id"], emailid=data["emailid"])
+        return cls(username=data["username"], id=data["id"], emailid=data["emailid"],
+                   fullname=data.get("fullname") or "")
 
 
 @dataclass
@@ -1141,30 +1141,86 @@ class IdpLoginAuthResponse:
 
     The success path returns via a bare json.dumps (text/html Content-Type,
     same pattern as create_operation/get_operation_by_id) with
-    {"success": True, "token": ..., "user": {...}}. Every failure path
-    (invalid/expired token, no matching user, or a malformed request
-    raising TypeError) returns via flask.jsonify (application/json) with
-    just {"success": False} and HTTP 401 -- to_dict() lets the handler keep
-    using jsonify() for that path without silently changing its
-    Content-Type, mirroring DeleteOperationResponse's to_dict()/to_text()
-    split.
+    {"success": True, "token": ..., "user": {...}}; token is the login token,
+    the same as the one of POST endpoints.TOKEN. Every failure path returns
+    via flask.jsonify (application/json) with {"success": False}: HTTP 401
+    for an invalid, expired or used code, HTTP 409 with link_required and
+    the emailid of the account if the SAML identity isn't linked to an
+    account yet and its email belongs to one -- the user logs in to that
+    account and links the identity by POST endpoints.IDP_LINK with the same
+    code. HTTP 403 with a message for the user if the login is refused,
+    e.g. for an account whose email isn't confirmed. to_dict() lets the
+    handler keep using jsonify() for these paths.
+
+    Servers before 12.0.0 answer success with a one-time password in token;
+    a 12.0.0 server marks its login token with "login_token": true.
+    legacy_password, a success without that mark, tells msui to log in with
+    the password at endpoints.TOKEN, as the older servers expect.
     """
 
     success: bool
     token: Optional[str] = None
     user: Optional[IdpUserInfo] = None
+    link_required: bool = False
+    emailid: Optional[str] = None
+    message: Optional[str] = None
+    legacy_password: bool = False
 
     def to_text(self):
-        return json.dumps({"success": self.success, "token": self.token, "user": self.user.to_dict()})
+        return json.dumps({"success": self.success, "token": self.token, "login_token": True,
+                           "user": self.user.to_dict()})
 
     def to_dict(self):
-        return {"success": self.success}
+        data = {"success": self.success}
+        if self.link_required:
+            data.update(link_required=True, emailid=self.emailid)
+        if self.message is not None:
+            data.update(message=self.message)
+        return data
 
     @classmethod
     def from_text(cls, text):
         data = json.loads(text)
-        user = IdpUserInfo.from_dict(data["user"]) if "user" in data else None
-        return cls(success=data["success"], token=data.get("token"), user=user)
+        user = IdpUserInfo.from_dict(data["user"]) if data.get("user") else None
+        return cls(success=data["success"], token=data.get("token"), user=user,
+                   link_required=bool(data.get("link_required", False)), emailid=data.get("emailid"),
+                   message=data.get("message"),
+                   legacy_password=bool(data.get("success")) and data.get("login_token") is not True)
+
+
+@dataclass
+class IdpLinkRequest:
+    """POST endpoints.IDP_LINK -- link the SAML identity of a one-time code,
+    which endpoints.IDP_LOGIN_AUTH refused with link_required, to the logged
+    in user (token is the login token, as for every other route).
+    """
+
+    token: str
+    code: str
+
+    def to_form_data(self):
+        return {"token": self.token, "code": self.code}
+
+    @classmethod
+    def from_form(cls, form):
+        return cls(token=form.get("token"), code=form.get("code"))
+
+
+@dataclass
+class IdpLinkResponse:
+    """Response body for POST endpoints.IDP_LINK (or the body "False" for an
+    invalid login token, as for every route of verify_user)."""
+
+    success: bool
+
+    def to_dict(self):
+        return {"success": self.success}
+
+    @classmethod
+    def from_text(cls, text):
+        if text == AUTH_FAILED_TEXT:
+            return cls(success=False)
+        return cls(success=bool(json.loads(text).get("success")))
 
 
 @dataclass

mslib/mscolab/app/__init__.py

@@ -271,9 +271,10 @@ def repair_schema():
     """
     from alembic.migration import MigrationContext
     from alembic.operations import Operations
-    from mslib.mscolab.migrations.schema_repair import (AUTOINCREMENT_TABLES, add_fullname, add_token_nonce,
-                                                        fix_postgresql_sequences, has_autoincrement_ids,
-                                                        has_column, has_dangling_references,
+    from mslib.mscolab.migrations.schema_repair import (AUTOINCREMENT_TABLES, add_fullname, add_saml_tables,
+                                                        add_token_nonce, fix_postgresql_sequences,
+                                                        has_autoincrement_ids, has_column,
+                                                        has_dangling_references, has_table,
                                                         sqlite_foreign_keys_off, use_autoincrement_ids)
     from mslib.mscolab.models import db
 
@@ -294,6 +295,10 @@ def repair_schema():
             logging.warning("The database misses users.token_nonce, it ran an earlier development state of the "
                             "12.0.0 migration. Adding it now; every user has to log in again.")
             add_token_nonce(Operations(MigrationContext.configure(conn)), conn)
+        if not (has_table(conn, "idp_identities") and has_table(conn, "saml_logins")):
+            logging.warning("The database misses the tables of SAML logins, it ran an earlier development state of "
+                            "the 12.0.0 migration. Adding them now.")
+            add_saml_tables(Operations(MigrationContext.configure(conn)), conn)
         if has_dangling_references(conn):
             # Only a database that ran an earlier development state of the 12.0.0 migration, which repairs them:
             # a server start doesn't delete rows. Until they are repaired, deleting their user or operation fails.