diff --git a/CHANGELOG.md b/CHANGELOG.md index f8c512f..43a61cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,9 @@ The notable changes of every release. The section of a version is the text of it older than the lease (`easyssf.receiver.dedup.lease`, 60 seconds) counts as abandoned and the SET is handled again. +- `SsfEventToken.idempotencyKey()` and `SsfEventContext.idempotencyKey()`, the issuer and `jti` of a + SET, as the key for the side effects of a handler: processing is at least once, which the + documentation now says in those words. - Schema migration: `easyssf-receiver-jdbc` ships versioned migration scripts in `org/easyssf/receiver/jdbc/migration/` (Flyway naming, plain SQL), and `JdbcSsfSchema.prepareTable` adds the columns a release introduced to an existing table where it may create tables diff --git a/README.md b/README.md index eae72c6..87bbe16 100644 --- a/README.md +++ b/README.md @@ -157,10 +157,14 @@ class StepUpHandler implements SsfEventHandler { } ``` -Handlers must be idempotent. If a handler throws, the SET is not acknowledged and the transmitter is -expected to deliver it again, in which case all handlers run again, possibly on another instance of -the application: with a shared dedup store, a SET is claimed while its handlers run, another instance -leaves it alone meanwhile, and a claim a crashed instance left behind is taken over after a lease. +**A SET is handled at least once, not exactly once**, so handlers must be idempotent. If a handler +throws, the SET is not acknowledged and the transmitter is expected to deliver it again, in which case +all handlers run again, possibly on another instance of the application. With a shared dedup store a +SET is claimed while its handlers run and another instance leaves it alone meanwhile, and a redelivery +of a handled SET is skipped; but an instance that dies after a handler's side effects and before the +success is recorded hands the SET to the next instance after a lease. For side effects outside the +receiver's stores, a database row, a message, an email, key them by `eventContext.idempotencyKey()`, +the issuer and `jti` of the SET. `SsfSubject` is the `sub_id` of the SET: a simple subject identifier (RFC 9493: `iss_sub`, `email`, `opaque`, `account`, `phone_number`, `did`, `uri`, `aliases`; RFC 9967: `scim`) or a complex subject whose members diff --git a/easyssf-core/src/main/java/org/easyssf/core/event/SsfEventToken.java b/easyssf-core/src/main/java/org/easyssf/core/event/SsfEventToken.java index b2f31ff..a070a62 100644 --- a/easyssf-core/src/main/java/org/easyssf/core/event/SsfEventToken.java +++ b/easyssf-core/src/main/java/org/easyssf/core/event/SsfEventToken.java @@ -28,4 +28,15 @@ public record SsfEventToken(String jti, String iss, Instant iat, List au claims = SsfCollections.copyOf(claims); } + /** + * The key that identifies this SET across deliveries: the issuer and the {@code jti}, + * which is only unique per issuer (RFC 8417, section 2.2). A handler that has side + * effects outside the receiver's stores uses it to make them idempotent, because a + * SET may be handled more than once: after a handler failed, or on another instance + * after the one that handled it died before recording the success. + */ + public String idempotencyKey() { + return this.iss + "::" + this.jti; + } + } diff --git a/easyssf-receiver-jdbc/AGENTS.md b/easyssf-receiver-jdbc/AGENTS.md index 1a071ff..21dda8e 100644 --- a/easyssf-receiver-jdbc/AGENTS.md +++ b/easyssf-receiver-jdbc/AGENTS.md @@ -16,8 +16,8 @@ The JDBC stores of the receiver in plain SQL, over the small `SsfJdbcOperations` `./mvnw -pl easyssf-receiver-jdbc -Pdatabase-tests verify` (Docker). Run them when the SQL or the schema changes. All share `AbstractJdbcSsfStoresTests`; add store tests there. `AbstractJdbcSharedReceiversTests` (same three databases) runs two receivers against one - `TestTransmitter` on shared stores: a SET handled once, acknowledgements handed over between - instances, and the race where one instance acknowledges a SET another is still handling (covered + `TestTransmitter` on shared stores: a SET handled by one instance and skipped by the others, + acknowledgements handed over between instances, and the race where one instance acknowledges a SET another is still handling (covered by the in-progress state of the dedup store, see `scratch/plans/dedup-in-progress-state.md`). - The module is an explicit Java module; test compilation adds `java.naming` because H2's data source implements a `javax.naming` interface (see `pom.xml`). Keep that if you touch the diff --git a/easyssf-receiver-jdbc/src/main/java/org/easyssf/receiver/jdbc/JdbcSsfJtiDedupStore.java b/easyssf-receiver-jdbc/src/main/java/org/easyssf/receiver/jdbc/JdbcSsfJtiDedupStore.java index 857cb5a..e077bcd 100644 --- a/easyssf-receiver-jdbc/src/main/java/org/easyssf/receiver/jdbc/JdbcSsfJtiDedupStore.java +++ b/easyssf-receiver-jdbc/src/main/java/org/easyssf/receiver/jdbc/JdbcSsfJtiDedupStore.java @@ -10,9 +10,11 @@ import org.easyssf.receiver.set.SsfJtiDedupStore; /** - * {@link SsfJtiDedupStore} that remembers processed SETs in a database table, so that a - * SET is handled once even if it is delivered to several instances of an application or - * again after a restart. + * {@link SsfJtiDedupStore} that remembers processed SETs in a database table shared by + * the instances of an application: a SET delivered to several of them or again after a + * restart is handled by one and skipped by the others, and a SET one instance is handling + * is left alone by the rest. Processing stays at least once, see + * {@link SsfJtiDedupStore}. * *

* A claim inserts the row with the state {@code IN_PROGRESS}: the primary key of the diff --git a/easyssf-receiver-jdbc/src/test/java/org/easyssf/receiver/jdbc/AbstractJdbcSharedReceiversTests.java b/easyssf-receiver-jdbc/src/test/java/org/easyssf/receiver/jdbc/AbstractJdbcSharedReceiversTests.java index 6d80730..9616d16 100644 --- a/easyssf-receiver-jdbc/src/test/java/org/easyssf/receiver/jdbc/AbstractJdbcSharedReceiversTests.java +++ b/easyssf-receiver-jdbc/src/test/java/org/easyssf/receiver/jdbc/AbstractJdbcSharedReceiversTests.java @@ -28,8 +28,8 @@ /** * Several receivers (instances of one application) polling the same stream and sharing - * the JDBC stores: a SET is handled once, and an acknowledgement one instance could not - * deliver leaves with the next poll of another. + * the JDBC stores: a SET is handled by one of them and skipped by the others, and an + * acknowledgement one instance could not deliver leaves with the next poll of another. */ abstract class AbstractJdbcSharedReceiversTests { diff --git a/easyssf-receiver-spring-boot-starter/README.md b/easyssf-receiver-spring-boot-starter/README.md index 2134a5f..2a78078 100644 --- a/easyssf-receiver-spring-boot-starter/README.md +++ b/easyssf-receiver-spring-boot-starter/README.md @@ -119,8 +119,11 @@ Things to know: **SCIM Events** (RFC 9967, see the [root README](../README.md#scim-events)): the deactivation and the deletion of a SCIM user usually mean the user may no longer be logged in. To end their local -sessions, add the events to the user events. A `scim` subject is matched with the logged-in user by -comparing attributes of the resource with claims: by default `externalId` and `id` (the `id` +sessions, add the events to the user events. Which SCIM resource is which logged-in user is not +something SCIM or OpenID Connect define: it is a decision of your deployment, about identifiers the +SCIM service provider and the OpenID Provider happen to share, and the mapping below is where you +state it. A `scim` subject is matched with the logged-in user by comparing attributes of the resource +with claims: by default `externalId` and `id` (the `id` attribute, or the last segment of the `uri`) with `sub`, and `userName` with `preferred_username`. If your SCIM service provider and your OpenID Provider share other identifiers, configure the pairs; a multi-valued attribute such as `emails` matches if any of its values does, and values compared with @@ -331,8 +334,10 @@ the state is kept in its database instead, without further configuration: instance that receives a SET another one is handling leaves it for the transmitter's redelivery, neither handling nor acknowledging it, so a handler failure on the first instance is not masked by an acknowledgement of the second. A claim older than `easyssf.receiver.dedup.lease` (60 seconds) - counts as abandoned by a crashed instance and the SET is handled again, which is one more reason for - handlers to be idempotent. Set the lease longer than your longest handler. + counts as abandoned by a crashed instance and the SET is handled again. The guarantee is therefore + at-least-once processing with concurrent deliveries suppressed, not exactly once: handlers stay + idempotent and key their side effects by `eventContext.idempotencyKey()`. Set the lease longer than + your longest handler. - **Upgrading from 0.1.0 or 0.2.0**: run `migration/V0_3_0__dedup_state_and_poll_acks.sql`, which adds the column `STATE` to `EASYSSF_PROCESSED_SET` and creates `EASYSSF_POLL_ACK`; with `initialize-schema` `embedded` or `always` the receiver does it on startup. diff --git a/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventContext.java b/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventContext.java index 9a793f2..e202a2c 100644 --- a/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventContext.java +++ b/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventContext.java @@ -25,6 +25,15 @@ public SsfEventToken eventToken() { return this.eventToken; } + /** + * The key for making the side effects of a handler idempotent, the issuer and + * {@code jti} of the SET: a SET is handled at least once, not exactly once (see + * {@link SsfEventHandler}). + */ + public String idempotencyKey() { + return this.eventToken.idempotencyKey(); + } + /** * The event type URIs contained in the SET. */ diff --git a/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventHandler.java b/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventHandler.java index 0bff19d..8482aeb 100644 --- a/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventHandler.java +++ b/easyssf-receiver/src/main/java/org/easyssf/receiver/event/SsfEventHandler.java @@ -5,8 +5,13 @@ * {@link org.easyssf.receiver.set.SsfSetProcessor} is invoked for every SET. * *

- * Handlers must be idempotent: a SET is delivered again if any handler failed, and - * de-duplication does not survive a restart. + * A SET is handled at least once, not exactly once, so handlers must be idempotent. A SET + * is delivered again if any handler failed, and all handlers run again, possibly on + * another instance of the application; with a shared dedup store a SET is claimed while + * its handlers run and another instance leaves it alone meanwhile, but an instance that + * dies after a handler's side effects and before the success is recorded hands the SET to + * the next instance after a lease. Side effects outside the receiver's stores should be + * keyed by {@link SsfEventContext#idempotencyKey()}. */ @FunctionalInterface public interface SsfEventHandler { diff --git a/easyssf-receiver/src/main/java/org/easyssf/receiver/set/SsfJtiDedupStore.java b/easyssf-receiver/src/main/java/org/easyssf/receiver/set/SsfJtiDedupStore.java index a65be3b..e428863 100644 --- a/easyssf-receiver/src/main/java/org/easyssf/receiver/set/SsfJtiDedupStore.java +++ b/easyssf-receiver/src/main/java/org/easyssf/receiver/set/SsfJtiDedupStore.java @@ -3,8 +3,11 @@ import org.easyssf.core.event.SsfEventToken; /** - * Remembers which SETs were already processed, so that a SET delivered more than once is - * handled only once. Implement it to back it with a shared or durable store. + * Remembers which SETs were processed, so that a redelivery of a handled SET is skipped + * and a SET another instance is handling is left alone. It does not make handling exactly + * once: an instance that dies after a handler ran and before {@link #processed} hands the + * SET to the next instance after the lease, which is why handlers are idempotent. + * Implement it to back it with a shared or durable store. * *

* A SET is {@link #claim claimed} before its handlers run and marked {@link #processed @@ -68,11 +71,11 @@ default boolean seenBefore(SsfEventToken eventToken) { } /** - * The key identifying a SET. A {@code jti} is only unique per issuer (RFC 8417, - * section 2.2). + * The key identifying a SET, the same one handlers use for their side effects. + * @see SsfEventToken#idempotencyKey() */ static String key(SsfEventToken eventToken) { - return eventToken.iss() + "::" + eventToken.jti(); + return eventToken.idempotencyKey(); } } diff --git a/easyssf-receiver/src/test/java/org/easyssf/receiver/event/SsfEventContextTests.java b/easyssf-receiver/src/test/java/org/easyssf/receiver/event/SsfEventContextTests.java index 5be4b01..7e42857 100644 --- a/easyssf-receiver/src/test/java/org/easyssf/receiver/event/SsfEventContextTests.java +++ b/easyssf-receiver/src/test/java/org/easyssf/receiver/event/SsfEventContextTests.java @@ -34,6 +34,13 @@ void vendorEventTypesAreAddressedByUri() { assertThat(SsfEventTypes.aliasOf(vendorType)).isEqualTo(vendorType); } + @Test + void idempotencyKeyIsIssuerAndJti() { + SsfEventContext context = context(Map.of(SsfEventTypes.CAEP_SESSION_REVOKED, Map.of()), null); + assertThat(context.idempotencyKey()).isEqualTo("https://idp.example::jti-1"); + assertThat(context.idempotencyKey()).isEqualTo(context.eventToken().idempotencyKey()); + } + @Test void subjectIsTakenFromSubId() { SsfEventContext context = context(Map.of(SsfEventTypes.CAEP_SESSION_REVOKED, Map.of()), opaque("abc"));