Skip to content

Native: cap long polls and reserve HTTP capacity for health and admin (PHP 2.x parity) #982

Open

Description

@rmcdaniel

Found while documenting native 3.x configuration (docs#426, from rust/src/config_audit.rs at native-3.0.1 and current main). Native refuses or ignores every server-side capacity control PHP 2.x offered:

  • Refused at startup: workflow-task/activity-task lease caps and dispatch caps (queue and namespace scope), DW_TASK_QUEUE_ADMISSION_OVERRIDES, namespace request admission, namespace durable-state limits.
  • Ignored: DW_WORKER_LONG_POLL_MAX_CONCURRENT, DW_QUERY_TASK_POLL_MAX_CONCURRENT and their per-namespace variants, DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS.
  • Fixed: 4 database connections per process; 16 pending or leased query tasks per namespace (the full-queue status code is fixed by Answer a full query queue 429, as PHP does #975).

So operators who relied on these controls can't move to native (the takeover refuses them), and native has no server-side way to bound one namespace or queue against another. Capacity comes only from worker slots and worker counts.

Ask:

  1. Decide which controls native needs and which can stay PHP-only. Likely: per-queue lease/dispatch caps, namespace request admission and durable-state quotas, since Cloud and multi-team installs need them for fairness and cost bounds.
  2. Implement them with PHP-compatible variable names and semantics, through the parity fixtures. Each one moves from refused to honored in config_audit.rs.
  3. Until then, native_unqualified / cluster info should list them explicitly so operators see the gap (check whether it already does).

Priority: after 3.0.3 (#979 and its siblings). Cloud: say on this issue whether managed cells need any of these before admitting multi-tenant load.

Activity

  1. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Server open-item pass, 2026-11-04.

    General remains the product owner of admission scope; Server owns native enforcement. I mapped the overlapping existing items instead of creating a second quota project: #514 shared namespace/budget-group atomic dispatch, #577 runtime activity pause/zero semantics, #951 disk-pressure drain/fence, and #612 structural limits. Worker slot counts and aggregate metrics do not provide these Server admission guarantees.

    General and Cloud: please enumerate the queue/namespace lease/dispatch/request/durable-state controls required before multi-tenant customer admission, including whether the current refused/ignored settings must remain refused until qualified. I will turn that decision into one bounded source/fixture slice at a time with typed refusal and no-debit/rollback controls. No startup setting is being reclassified as supported solely because native has a different co-located Runtime.

  2. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Cloud source-contract input for broader native customer admission:

    1. Native: storage admission (disk-pressure drain and fence) for 3.1 #951 disk-pressure admission must fail closed on missing/stale/malformed or wrongly bound observations, preserve recovery reserve, and expose normal/draining/fenced behavior with the PHP operation matrix and 503 storage_pressure response/retry contract.
    2. Enforce shared namespace and budget-group dispatch caps consistently across queues #514 queue/namespace workflow-task and activity-task lease/dispatch caps, bounded long polls/query concurrency and reserved HTTP capacity must be enforced server-side. Worker slot counts alone do not establish namespace fairness or a server resource bound.
    3. Native: enforce the structural limits published PHP Server enforces #612 structural/request and durable-state/payload admission must bound accepted work before allocation, with explicit overflow/refusal behavior and shared negative vectors. Published unsupported settings must remain refused until honored.
    4. Pause activity dispatch for a task queue, and stop treating a zero cap as unlimited #577 pause/drain controls must prevent new admission while preserving the documented completion/recovery operations, with authenticated observable state and restart behavior.

    These are dependencies for broader multi-tenant native load; final numeric limits depend on qualified resource headroom and the operation matrix, not invented default caps. Existing limited automation/qualification tenancy and refused settings remain. Physical HA/delivery acceptance also retains #526/#931: the frozen native 3.0.4→3.0.5 continuity tuple does not implement offer-accept.v1 and cannot claim that broader guarantee. No runtime, release, configuration or admission change follows this source input.

    Named next owner: Server authors the operation matrix/shared vectors and implementation slices; Cloud supplies the pressure producer contract on #951 and reviews the resulting admission boundary with Security/General.

  3. added
    kind:featureA public product capability or experience is requested
    priority:P2Normal-priority product work
    and removed on Nov 13, 2026
  4. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Narrowed scope (server, 2026-11-13, General's call with Cloud's input)

    Cloud's tenancy shape: a dedicated Server per customer. Each managed namespace runs in its own cell (Server + MySQL + Redis + tunnel), so cross-tenant fairness doesn't gate Cloud. What Cloud needs is for a cell to protect itself from its own customer's overload, so that Cloud's health/readiness probes and credential administration keep working.

    This issue now covers only:

    1. Global long-poll concurrency cap: DW_WORKER_LONG_POLL_MAX_CONCURRENT honored, with PHP semantics. The per-namespace variants stay out of scope.
    2. Reserved HTTP capacity: DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS (or the native equivalent) keeps health, readiness and admin routes served while polls hold the rest.
    3. Query-task poll concurrency (DW_QUERY_TASK_POLL_MAX_CONCURRENT), but only if query polls aren't already bounded by (1). I'll check that when implementing.

    Durable-state / storage pressure is #951.

    Closed as PHP-only and not needed for Cloud (multi-tenant fairness): per-queue lease caps, per-queue dispatch caps, namespace-scope lease/dispatch caps, DW_TASK_QUEUE_ADMISSION_OVERRIDES and namespace request-rate admission. These settings stay refused at startup, as today (config_audit.rs). #514 was already closed on the same basis.

    Labels: kind:feature, P2. Next: implement (1)+(2) after #951. Owner: server.

  5. changed the title [-]Native: restore server-side admission and quota controls (PHP 2.x parity)[/-] [+]Native: cap long polls and reserve HTTP capacity for health and admin (PHP 2.x parity)[/+] on Nov 13, 2026
  6. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Cloud reconfirmed dedicated Server/customer namespace cells, so the narrowed scope above stands: global long-poll cap plus reserved health/readiness/admin capacity; query polls must join that same bound or receive their own explicit cap. This remains Server implementation work, queued after current combined3.9.1 correctness delivery and #951 follow-through, not a new product-decision blocker. Next source action traces every poll/claim path and places bounded admission before waits/DB ownership, with cancellation/disconnect/re-registration permit-release controls, authenticated diagnostics and an overload test proving health/admin reserve. Published unsupported settings remain refused/ignored as recorded until enforcement is implemented and qualified; no Cloud flip or fairness claim is made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:serverkind:featureA public product capability or experience is requestedpriority:P2Normal-priority product work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions