Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
0bd3078
Add subscription partitions with a routing inbox
DavidBadura Oct 14, 2026
d4a8e95
Read only the inbox of a partition and claim by pending work
DavidBadura Oct 14, 2026
b493604
Claim subscriptions in groups and batch the inbox queries
DavidBadura Oct 14, 2026
3c33500
Insert and update subscriptions in batches
DavidBadura Oct 14, 2026
24acd43
Never let the batches of a claimed group overlap
DavidBadura Oct 15, 2026
7520d0a
Only group subscriptions with close positions
DavidBadura Oct 15, 2026
20b1bf9
Clear the processed messages of a failed partition from its inbox
DavidBadura Oct 15, 2026
ccbc05f
Require PHP 8.4
DavidBadura Oct 15, 2026
055d49b
Load error context and cleanup tasks lazily for engine lookups
DavidBadura Oct 15, 2026
21b43b5
Keep the subscription store interface small
DavidBadura Oct 15, 2026
29a5ced
Keep MessageLoader and the claim API small
DavidBadura Oct 15, 2026
f74cf53
Stop a partitioned subscriber on a systemic failure
DavidBadura Oct 15, 2026
ee14631
Add #[PartitionKey] to resolve the key of the processing partition
DavidBadura Oct 15, 2026
829dbe5
Load the inbox messages of a group in chunks
DavidBadura Oct 15, 2026
6909233
Move a partition past inbox messages that no longer exist
DavidBadura Oct 15, 2026
255def0
Keep the boot of partitioned subscribers linear
DavidBadura Oct 15, 2026
e929d78
Look up the partitions of a run together
DavidBadura Oct 15, 2026
8963e57
Process the routed messages in the same run
DavidBadura Oct 15, 2026
3ce0006
Only set back partitions when the circuit breaker stopped the subscriber
DavidBadura Oct 15, 2026
22c72ea
Update the circuit breaker note about #[OnFailed]
DavidBadura Oct 15, 2026
e03bf6c
Compare subscription ids binary on MySQL and MariaDB
DavidBadura Oct 15, 2026
4af31f4
Stop a subscription for the rest of the run instead of skipping a mes…
DavidBadura Oct 15, 2026
c2370ae
Detach every partition of a detached subscriber
DavidBadura Oct 15, 2026
4ba4952
Only remove partitions when the main subscription is removed
DavidBadura Oct 15, 2026
6ab608d
Document that booting partitions do not emit events
DavidBadura Oct 15, 2026
252801d
Explain the message limit of partition groups
DavidBadura Oct 15, 2026
fcee4e6
Read only the limit of every inbox
DavidBadura Oct 15, 2026
009ab9f
Document how batching works with partitions
DavidBadura Oct 15, 2026
88f23eb
Emit the events of all partitions into the stream of their subscriber
DavidBadura Oct 15, 2026
c2c621b
Pause and reactivate partitions in groups
DavidBadura Oct 15, 2026
c5c403f
Remove subscriptions with one statement per chunk
DavidBadura Oct 15, 2026
cca3d93
Correct the commit granularity of partition groups in the upgrade guide
DavidBadura Oct 15, 2026
9eaab6e
Remove idle partitions
DavidBadura Oct 15, 2026
4c096be
Finish the partitions of a run once subscriber
DavidBadura Oct 15, 2026
9d0dfa7
Test partitions with two real worker processes
DavidBadura Oct 15, 2026
f27ed67
Test batching and the event emitter with two workers
DavidBadura Oct 15, 2026
78ddded
Set the partitioner on the subscriber class
DavidBadura Oct 15, 2026
225f875
Add bucket, tag and hashed partitioners
DavidBadura Oct 15, 2026
cbb7c0e
Delete processed inbox rows by their key
DavidBadura Oct 15, 2026
22e37a9
Format the partition key example in the docs
DavidBadura Oct 15, 2026
5fb8362
Keep the partitioner in the subscriber metadata
DavidBadura Oct 15, 2026
49f53a2
Load only the partitions that need attention in the status command
DavidBadura Oct 15, 2026
fe390aa
Count subscriptions per status with stats()
DavidBadura Oct 15, 2026
b7cce29
Rely on the positive-int type for the number of buckets
DavidBadura Oct 15, 2026
1c84d0a
Wait for locked subscriptions in commands
DavidBadura Oct 15, 2026
aa7f57b
Never start a partition in the error status of its main subscription
DavidBadura Oct 15, 2026
a355edf
Keep failures of one subscription out of its group
DavidBadura Oct 15, 2026
3ba93a1
Store nothing about a held back failure before the run ends
DavidBadura Oct 15, 2026
b0df88b
Drop held back failures whenever the circuit breaker trips
DavidBadura Oct 15, 2026
ba086e3
Decide a stop before OnHandleMessage
DavidBadura Oct 15, 2026
d68c955
Find main subscriptions without reading their partitions
DavidBadura Oct 15, 2026
fa2aaeb
Boot partitions completely without a limit
DavidBadura Oct 15, 2026
fdda557
Count the partitions left to finish instead of loading them
DavidBadura Oct 15, 2026
e57cd5d
Match the tag prefix of TagPartitioner like EventTag does
DavidBadura Oct 15, 2026
98afc20
Reject subscriber ids that collide with partition ids
DavidBadura Oct 15, 2026
212f7cd
Reserve # for partition ids in every subscriber id
DavidBadura Oct 15, 2026
ad471ca
Count the messages waiting in the inboxes of the partitions
DavidBadura Oct 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/unit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ jobs:
- "lowest"
- "highest"
php-version:
- "8.2"
- "8.3"
- "8.4"
- "8.5"
operating-system:
Expand Down
4 changes: 2 additions & 2 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
}
],
"require": {
"php": "~8.2.0 || ~8.3.0 || ~8.4.0 || ~8.5.0",
"php": "~8.4.0 || ~8.5.0",
"doctrine/dbal": "^4.4.0",
"doctrine/migrations": "^3.3.2",
"patchlevel/hydrator": "^2.0.2",
Expand All @@ -48,7 +48,7 @@
"symfony/type-info": "^7.2.0 || ^8.0.0"
},
"require-dev": {
"ext-pdo_sqlite": "~8.2.0 || ~8.3.0 || ~8.4.0 || ~8.5.0",
"ext-pdo_sqlite": "~8.4.0 || ~8.5.0",
"doctrine/orm": "^2.18.0 || ^3.0.0",
"infection/infection": "^0.32.0",
"league/commonmark": "^2.6.1",
Expand Down
6 changes: 3 additions & 3 deletions composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

87 changes: 78 additions & 9 deletions docs/UPGRADE-4.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ searchable: false
---
# Upgrade 4.0

## PHP

The minimum PHP version is now 8.4.

## Aggregates

### Aggregate Root
Expand Down Expand Up @@ -423,22 +427,37 @@ This is mostly transparent, but a few contracts changed.
#### SubscriptionStore

`claim()` and `inLock()` are now mandatory parts of the `SubscriptionStore` interface, and the
separate `LockableSubscriptionStore` interface has been removed. Every custom store has to implement
both:
separate `LockableSubscriptionStore` interface has been removed. `add()`, `update()` and `remove()` take
several subscriptions at once. Every custom store has to implement:

```php
use Patchlevel\EventSourcing\Subscription\Store\SubscriptionCriteria;
use Patchlevel\EventSourcing\Subscription\Store\SubscriptionStats;
use Patchlevel\EventSourcing\Subscription\Subscription;

interface SubscriptionStore
{
// ... get/find/add/update/remove ...
// ... get/find ...

/** Count the matching subscriptions per status, without loading them. */
public function stats(SubscriptionCriteria $criteria): SubscriptionStats;

/**
* Claim exactly one subscription via a row lock (SKIP LOCKED). Return null if the row is held
* by another worker or no longer matches the criteria.
* Claim the subscriptions via row locks. Leave out the ones that no longer match the criteria,
* and the ones held by another worker (SKIP LOCKED). With $wait, wait for them instead,
* commands like pause or remove have to reach every subscription.
*
* @param non-empty-list<string> $ids
*
* @return list<Subscription>
*/
public function claim(string $id, SubscriptionCriteria $criteria): Subscription|null;
public function claim(array $ids, SubscriptionCriteria $criteria, bool $wait = false): array;

public function add(Subscription ...$subscriptions): void;

public function update(Subscription ...$subscriptions): void;

public function remove(Subscription ...$subscriptions): void;

/**
* @param Closure():T $closure
Expand All @@ -453,14 +472,22 @@ interface SubscriptionStore
`find()` no longer locks the matched rows: it is now a plain, unlocked snapshot read. The locking
happens per subscription inside `claim()`.

The `SubscriptionEngine` interface has the new method `stats()` as well, a custom engine or decorator passes it on.

The `DoctrineSubscriptionStore` loads the error context and the cleanup tasks of the subscriptions from `find()`
lazily. Subscriptions that have one of them are native lazy objects, which `var_dump()` and `assertEquals()` show
without these fields until they are accessed.

#### Message limit is now per subscription

For `Run` and `Boot`, the `limit` used to cap the total number of messages across the shared stream.
Because there is no shared stream anymore, `limit` now caps the messages **per subscription**. One
`subscription:run` pass therefore processes up to `limit × number of subscriptions` messages. The
CLI default of `message-limit=100` now means "100 per subscription". It also defines the
checkpoint/lock-hold granularity: a unit of at most `limit` messages commits atomically and releases
the lock afterwards.
CLI default of `message-limit=100` now means "100 per subscription".
It also defines the checkpoint/lock-hold granularity: a unit of at most `limit` messages commits atomically and
releases the lock afterwards. If several subscriptions are claimed together with `subscriptionsPerClaim`, the limit
applies to the whole group. Partitions claimed together with `partitionsPerClaim` each process up to `limit`
messages, so such a group commits up to `partitionsPerClaim × limit` messages at once.

#### Error isolation

Expand All @@ -470,6 +497,48 @@ are logged and retried on the next pass without landing in `result.errors`. Any
that single subscription into status `Error` and surfaces in `result.errors`, while the remaining
subscriptions keep processing.

#### Subscription partitions

Subscribers can be split into partitions with `#[Partition]`, see the subscription docs.
A subscriber id must no longer contain `#`, it is reserved for the ids of the partitions.
The subscription table gets the columns `subscriber_id` and `partition_key`, and there is a new table
`subscriptions_inbox` (or `{table name}_inbox`). Since `subscriber_id` is not nullable, fill it before
making it `NOT NULL`, otherwise the migration fails on existing rows:

```sql
ALTER TABLE subscriptions ADD subscriber_id VARCHAR(255) DEFAULT NULL;
ALTER TABLE subscriptions ADD partition_key VARCHAR(255) DEFAULT NULL;
UPDATE subscriptions SET subscriber_id = id;
ALTER TABLE subscriptions ALTER COLUMN subscriber_id SET NOT NULL;
CREATE INDEX IDX_SUBSCRIPTIONS_SUBSCRIBER_ID ON subscriptions (subscriber_id, status, last_saved_at);
CREATE INDEX IDX_SUBSCRIPTIONS_PARTITION_KEY ON subscriptions (partition_key, status);
CREATE TABLE subscriptions_inbox (subscription_id VARCHAR(255) NOT NULL, message_index INT NOT NULL, PRIMARY KEY (subscription_id, message_index));
```
On MySQL and MariaDB the ids and the partition key are compared binary, so that partition keys that only differ in
case are different partitions:

```sql
ALTER TABLE subscriptions ADD subscriber_id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL;
ALTER TABLE subscriptions ADD partition_key VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL;
UPDATE subscriptions SET subscriber_id = id;
ALTER TABLE subscriptions MODIFY subscriber_id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL;
ALTER TABLE subscriptions MODIFY id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL;
CREATE INDEX IDX_SUBSCRIPTIONS_SUBSCRIBER_ID ON subscriptions (subscriber_id, status, last_saved_at);
CREATE INDEX IDX_SUBSCRIPTIONS_PARTITION_KEY ON subscriptions (partition_key, status);
CREATE TABLE subscriptions_inbox (subscription_id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, message_index INT NOT NULL, PRIMARY KEY (subscription_id, message_index));
```
Adjust the table names if you configured your own.

A custom `SubscriptionStore` has to persist `partitionKey()` and `subscriberId()` and support the new
`partitions` and `pendingPartitionsOnly` criteria. To support partitions, it also has to implement the
`PartitionInboxStore` interface with `addToInbox()`, `inbox()`, `inboxStats()` and `removeIdlePartitions()`.
The inbox of a partition only holds messages after its position: updating a partition clears its inbox up to its position, removing it removes its inbox.
A store without partitions support throws `PartitionsNotSupported` as soon as a partitioned subscriber is processed.

The `ids` of `SubscriptionCriteria` now match the subscription
id or the subscriber id, so the id of a partitioned subscriber also selects its partitions.
For subscribers without partitions nothing changes, because the subscriber id is the subscription id.

#### No global ordering across subscriptions

Subscriptions are processed independently, so there is no global ordering of messages across
Expand Down
29 changes: 20 additions & 9 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,15 +170,17 @@ After adding a new subscriber, run the setup and boot step and restart the run c

The boot and run commands are workers and support the following options:

| Option | Description |
|-------------------------|------------------------------------------------------------------------------------|
| `--run-limit` | Stop the worker after this number of runs. |
| `--message-limit` | How many events are processed per run. |
| `--memory-limit` | Stop the worker if it uses more memory than this, e.g. `256M` or `250MB`. |
| `--time-limit` | Stop the worker after this number of seconds. |
| `--sleep` | How many milliseconds the worker waits between two runs. |
| `--restart-signal-file` | Stop the worker when this file is touched after it has started. |
| `--heartbeat-file` | Touch this file on start and after every run, and remove it when the worker stops. |
| Option | Description |
|-----------------------------|----------------------------------------------------------------------------------------------------------------|
| `--run-limit` | Stop the worker after this number of runs. |
| `--message-limit` | How many events are processed per run. |
| `--memory-limit` | Stop the worker if it uses more memory than this, e.g. `256M` or `250MB`. |
| `--time-limit` | Stop the worker after this number of seconds. |
| `--sleep` | How many milliseconds the worker waits between two runs. |
| `--restart-signal-file` | Stop the worker when this file is touched after it has started. |
| `--heartbeat-file` | Touch this file on start and after every run, and remove it when the worker stops. |
| `--subscriptions-per-claim` | How many subscriptions are processed together in one transaction (default `1`). |
| `--partitions-per-claim` | How many [partitions](subscription.md#partitioning) are processed together in one transaction (default `100`). |

```bash
bin/console event-sourcing:subscription:run --memory-limit=250MB --time-limit=3600
Expand Down Expand Up @@ -238,6 +240,15 @@ If the subscription has an error, the command also prints the error message and
```bash
bin/console event-sourcing:subscription:status profile_1
```
For a partitioned subscriber, the table shows the number of partitions, how many are not active
and how many messages wait in how many inboxes, e.g. `1204 (error: 2, booting: 5), inbox: 3500 in 120`.
The partitions are counted, not loaded.
Use `--partitions` to list them, and pass the id of a partition to show its details.

```bash
bin/console event-sourcing:subscription:status --partitions
bin/console event-sourcing:subscription:status 'profile_1#tenant-a'
```
### Pause

The `event-sourcing:subscription:pause` command pauses subscriptions.
Expand Down
8 changes: 8 additions & 0 deletions docs/store.md
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,14 @@ use Patchlevel\EventSourcing\Store\Criteria\StreamCriterion;
$criterion = new StreamCriterion('profile-*', 'hotel-*');
$criterion = StreamCriterion::startWith('profile-');
```
To load specific messages by their index, use the `IndexCriterion`.

```php
use Patchlevel\EventSourcing\Store\Criteria\Criteria;
use Patchlevel\EventSourcing\Store\Criteria\IndexCriterion;

$criteria = new Criteria(new IndexCriterion(12, 57, 103));
```
Or you can the criteria builder to create the criteria.

```php
Expand Down
Loading
Loading