feat(ratelimit)!: typed key_fn with validated payloads, payload_type required (#60) - #64
Open
rcbevans wants to merge 10 commits into
Open
feat(ratelimit)!: typed key_fn with validated payloads, payload_type required (#60)#64rcbevans wants to merge 10 commits into
rcbevans wants to merge 10 commits into
Conversation
rcbevans
force-pushed
the
spec/ratelimit-typing
branch
from
July 30, 2026 03:53
fbde9cd to
57cf49c
Compare
…yed refs Breaking change: key_fn now receives the validated Pydantic model, not the raw dict. Existing key_fn implementations that access p["key"] must be updated to p.attr. All existing test constructions migrated to .typed(). - KeyedRateLimitRef and KeyedReservationRef: payload_type is now required - .typed() classmethod provides compile-time type checking of key_fn - All test helpers (_rate_limit_ref, _keyed_ref) migrated to .typed() - All direct constructions across 12 test files migrated - e2e deliver_tenant_webhook actor migrated - New tests: .typed() construction, required payload_type, defaults, aliases, extra='forbid' propagation - KeyError tests updated to AttributeError (model attr access, not dict)
- All key_fn examples now use .typed() with model-attribute access - key_fn description updated: receives validated Pydantic model, not dict - Added .typed() usage examples with aliases and Field(alias=...) - Architecture: dispatch integration updated for validated_payload flow
…stry Add _resolve_key_fn_arg DRY helper that both _resolve_rate_limit_name and _resolve_reservation_name call to convert dict|BaseModel payloads into a validated BaseModel before passing to key_fn. Three cases: same type → zero-cost pass-through; different BaseModel → re-validate via model_dump()/model_validate(); raw dict → model_validate(dict). ValidationError from conversion propagates to the caller (payload error, not limiter fault). Widen payload types on acquire_for_actor and _validate_keyed_key to accept dict|BaseModel|None. Update wrong-model-type tests: previously expected AttributeError from key_fn accessing a missing field; now expects ValidationError from pydantic re-validation against ref.payload_type. Remove stale # type: ignore[arg-type] on composition test payloads that passed BaseModel to acquire_for_actor (now type-accepted).
Breaking change: _consumer.py now unconditionally passes the validated BaseModel (not the raw dict) to acquire_for_actor. The validated_payload fallback is moved before acquire so a ValidationError surfaces before a rate-limit token is consumed. - Moved validated_payload fallback before acquire_for_actor - Changed payload=job.payload to payload=validated_payload - Removed later redundant fallback (validated_payload always set) - Updated docstring to reflect pre-acquire validation - 2 new consumer tests: model passed to key_fn, validation before acquire - Updated existing test: validation failure no longer acquires resources - New e2e actor: deliver_typed_tenant_webhook with aliased TypedTenantPayload - New e2e test: typed keyed rate limit with aliases (both pass)
The test passed payload_type=BaseModel (uninstantiable) without
validated_payload. The old code skipped validation on the denied path
(bug: validation was after acquire). Now validation runs before acquire
— the test must pass validated_payload=EmptyPayload() to skip the
BaseModel.model_validate({}) call that would fail.
- Add payload_type=BaseModel guard (reject base class itself) - Fix no-op extra='forbid' tests to exercise through registry - Add isinstance assertions to typed ref resolution tests - Add ValidationError rollback test during partial composition - Add wrong-type-in-dict, nested model, extra='forbid' cross-type tests - Fix queue cap test: payload_type=EmptyPayload not BaseModel - Add consumer test: dict payload validated to model before acquire - Strengthen e2e test: split effects by tenant for independence proof - Fix pre-existing: wrap raw ValidationError as PayloadValidationError in dispatch and consumer paths (was classified as retryable)
- Add payload_type=BaseModel guard (reject base class itself) - Fix no-op extra='forbid' tests to exercise through registry - Add isinstance assertions to typed ref resolution tests - Add ValidationError rollback test during partial composition - Add wrong-type-in-dict, nested model, extra='forbid' cross-type tests - Fix queue cap test: payload_type=EmptyPayload not BaseModel - Add consumer test: dict payload validated to model before acquire - Strengthen e2e test: split effects by tenant for independence proof - Fix pre-existing: wrap raw ValidationError as PayloadValidationError in dispatch and consumer paths (was classified as retryable) - Update docs: ValidationError → PayloadValidationError (non-retryable)
…er, fix runner - H1: _resolve_key_fn_arg wraps ValidationError as PayloadValidationError (non-retryable) — was raw ValidationError, classified as retryable - M1: testing runner catches PayloadValidationError, routes to terminal failed write — was crashing with row stuck in running - M5: extracted validate_actor_payload helper to src/taskq/_validation.py, replaced 4 duplicated wrapping blocks across dispatch/consumer/jobs - M6: exc.errors(include_url=False, include_input=False) — no attacker- controlled field values in error messages persisted to DB/admin UI - C1: fixed pyright errors in negative-construction tests (attribute-free lambdas) - C2: ruff format on 3 test files
- M4: extracted _derive_keyed_key DRY helper; deleted duplicated blocks and stale comments; reverted _validate_keyed_key dead BaseModel arm - M7: fixed KeyedReservationRef docstring (row stores raw JSON, not model) - M8: documented key_fn type hole on both refs (.typed() + isinstance guard) - H2: added CHANGELOG [Unreleased] with breaking changes; added migration guide section to rate-limiting.md; fixed doc snippets missing imports - L1: removed stale # type: ignore[return-value] in registry.py - L2: removed stale # type: ignore[arg-type] in test_queue_concurrency_cap.py - L3: extracted _validate_concrete_payload_type shared helper in refs.py - L4: fixed .typed() docstrings (pass-through-or-revalidate, not just verify) - L5: fixed comments/docstrings: ValidationError -> PayloadValidationError - L6: e2e test asserts stored row carries wire alias tenantId; hoisted import - L7: added test for validated_payload=None short-circuit - L8: added assertions for PayloadValidationError structured attributes - Fixed: added payload column to _JOBS_COLUMNS in e2e test assertions
rcbevans
force-pushed
the
spec/ratelimit-typing
branch
from
July 30, 2026 04:59
57cf49c to
a5120da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
key_fnonKeyedRateLimitRefandKeyedReservationRefalways receive the validated Pydantic model instead of the rawdictfrom the job row.payload_typeis now a required field on both refs, a.typed()classmethod provides compile-time type safety, and the consumer unconditionally passes the validatedBaseModelto the registry — no raw-dict fallback path.Issue addressed
Closes #60 —
key_fnreceives the validated model, not the pre-validation wire dict.Implementation
payload_typeis now required. BothKeyedRateLimitRefandKeyedReservationRefdeclarepayload_type: type[BaseModel]with no default. Construction without it raisesValidationError. The field declares the model class the registry validates the payload against before callingkey_fn.key_fnsignature:Callable[[BaseModel], str]. The callable always receives the validated model instance — with Pydantic defaults, aliases, and validators applied. There is no untyped path;key_fnnever sees a raw dict..typed()classmethod on both refs: generic overP: BaseModel, acceptspayload_type: type[P]andkey_fn: Callable[[P], str]so the caller's lambda is type-checked against the concrete model's attributes at static-analysis time. The storedkey_fnfield is typed asCallable[[BaseModel], str](the registry guarantees the runtime type viaisinstance).RateLimitRegistry._resolve_key_fn_arg()— shared by_resolve_rate_limit_nameand_resolve_reservation_name. Three cases:isinstance(payload, ref.payload_type)isTrue): zero-cost pass-through — the exact same object is passed tokey_fn, no re-validation.BaseModeltype: re-validate viaref.payload_type.model_validate(payload.model_dump()). Converts the actor's model into the ref's declared model, applying the ref's aliases/defaults/validators. AValidationErrorfrom this surfaces as a payload error, not a limiter fault.ref.payload_type.model_validate(dict)._consumer.py— thevalidated_payloadfallback is moved beforeacquire_for_actorso the model is always available.payload=validated_payload(theBaseModel) is passed unconditionally — no gating, nojob.payload(raw dict) path. AValidationErrorfrom an invalid payload now surfaces before a rate-limit token is consumed.acquire_for_actorsignature:payloadparameter widened todict[str, object] | BaseModel | Noneto accept both forms (dict for direct callers, BaseModel for the dispatch path).Breaking changes
This is a breaking change — intentional for a pre-1.0 library heading to 1.0. No backward-compat shims.
payload_typeis required. Construction without it raisesValidationError. NoNonedefault, no optional.key_fnreceives aBaseModel, not adict. Existing implementations that access raw dict keys (p["tenantId"]) must be updated to model attributes (p.tenant_id). This is the point — the typed approach is strictly better: Pydantic defaults populate fields absent from the serialized payload, aliases map wire names to model attributes, and validation errors surface asValidationError(a payload error) rather thanKeyError/AttributeError(a limiter fault).ValidationErrorbeforeacquire_for_actoris called, so no rate-limit token is consumed on a job that will never run. Previously, the raw dict was passed tokey_fnand aKeyErrorcould occur after token acquisition.Test coverage
tests/test_ratelimit_refs.py.typed()classmethod: storespayload_type, construction withoutpayload_typeraisesValidationError,key_fnreceives model not dict,backendparameter, default fields, aliases,extra=forbidvalidation. Both refs.tests/test_ratelimit_keyed_rate_limits.py.typed(). Typed_resolve_rate_limit_name: dict→model validation, pydantic defaults applied, alias resolution, BaseModel same-type pass-through (identity check), BaseModel wrong-type re-validation,ValidationErrorpropagation,acquire_for_actorwithBaseModeland dict payloads.tests/test_ratelimit_keyed_refs.py.typed(). Typed_resolve_reservation_name: same coverage as rate limits, plus wrong-model-type raisesValidationError(notAttributeError).tests/test_keyed_reservation_hardening.py_keyed_refhelper migrated to.typed().tests/test_ratelimit_keyed_rate_limits_pg.py.typed().tests/test_ratelimit_keyed_refs_pg.py.typed().tests/test_consumer.pyBaseModel(not raw dict) toacquire_for_actor; validation failure before acquire means zero resources acquired;KeyedRateLimitRef.key_fnreceivesBaseModelat the consumer level.tests/test_ratelimit_composition.py.typed().tests/test_di_validate.py.typed().tests/test_worker_di_bootstrap.py.typed().tests/test_actor.py.typed().tests/test_queue_concurrency_cap.py.typed(); saturation test passes validated payload.tests/web_admin/test_ops.py.typed().tests/e2e/actors.pydeliver_tenant_webhookmigrated to.typed(). Newdeliver_typed_tenant_webhookactor withField(alias="tenantId")+serialize_by_alias=True— proves the validated model (not raw dict) reacheskey_fnend-to-end.tests/e2e/test_keyed_rate_limit.pytests/e2e/worker_entry.pyDocumentation
docs/guides/rate-limiting.md— allkey_fnexamples updated from dict access (p["tenant_id"]) to model attributes (p.tenant_id). Added.typed()usage examples with alias and default field demonstrations. Documentedpayload_typerequirement andValidationErrorvsKeyErrorsemantics.docs/architecture.md— dispatch integration section updated to describe validated payload model flow toacquire_for_actor.docs/specs/2026-07-29-ratelimit-typing.md— full design spec for this change.Closes #60