For the subnet operator: the person running the trainer, the funding intake, the provisioner and the public pages. Miners read MINER.md; validator operators read VALIDATOR.md; the system design is ARCHITECTURE.md.
The trusted orchestrator holds the wallet and private test pool, runs the
intake, and signs/publishes training records. Worker pods run training and
leave checkpoints for the orchestrator to collect. With v2 CPU shards, separate
CPU workers run the generator before GPU rental; older feeds run it on the pod.
Challenger legs bill the submitting miner's selected provider account. Under v2 king-ladder
rules the king normally funds its own leg too; failed-key recovery, configured
make-good capacity and validator evaluation use operator resources. The scheduler is
rolling intake + era king (DEC-CA-0043, cascade/trainer/rolling.py).
The implemented ladder supplies Cascade MoE Base variants at 17.58M,
69.02M, 273.84M and 1.23B total parameters. [v2.training] selects 1.23B on mainnet and 17.58M
for short testnet runs. The older flat-expert candidate remains available for replay.
The new source uses an explicit update budget; it does not assume a measured
GPU rate. Qualify the worker image, CPU generation image, baseline generator,
GPU memory/throughput and full benchmark pipeline before activation. 56M
remains unspecified and blocked.
- Before activation: pins, calibration and launch order
- Pods and the worker image: worker image,
GPU type, rented pods, operator lanes, sandbox,
cascade-provisioner - Funded intake: bring-up, fronting the intake, payer-pod credentials, bench, direct submissions, cost model
- Backlogs and recovery, including shared limits and operator-billed retries
- Security and credentials
- Dashboards and status: publishing, benchmark sources and missing-data checks
- C competition changes
- Network config overlays: change verdict / payout constants without a fleet release
Mainnet Lane A targets 30 October 2026, 12:00 UTC. With from_block = 0,
[v2] launch_at_utc resolves the first era boundary at or after the target
from finalized chain timestamps. A typed block is an explicit override. A
missing timestamp read leaves launch unresolved; it never estimates a block.
Nodes observing the crossing use recent timestamps; a much later fresh node
may need an archive-capable RPC to derive that same historical boundary. A
pruned node that cannot read it fails with an actionable error; point it at an
archive endpoint or type the resolved [v2] from_block.
The v2 paid-entry opening is separately pinned as [v2] commit_open_at_utc = "2026-10-30T09:00:00Z". Its first finalized chain block becomes
commit_from_block; an explicit block pin is also supported. A configured pin
does not open intake early: the finalized head must reach it first. Late nodes must
be able to resolve both timestamps, or use both published block pins. This is
also required after a restart. A running intake retains each resolved pin
between requests so it does not repeatedly need state the node has pruned.
The submission rule is an opening floor, not a closing deadline. The updated intake refuses paid A
entries before it opens and refuses older reveals afterward. The updated
trainer holds queued A challengers until v2 is active, without CPU/GPU rental
or consuming attempts. Installing this release therefore pauses new challenger
dispatch under v1; historical replay and king work remain available.
The planned order is A → C → B. C with takes_from = "lane_a" can launch with
B disabled or scheduled later. It reserves 25% while A keeps 75%; launching B
later yields 50%/25%/25%. Legacy C taken from B still requires B to be live.
Upgrade every validator and satellite to this release before arming C before B.
While launch_at_utc is set and from_block = 0, the trainer and validator
services refuse to start unless the v2 configuration the date would activate
loads (for example [netconfig] owner_hotkey for bench_source = "validator").
Complete that configuration or clear the date before deploying a release.
cascade-audit and satellites resolve the same launch from the chain
(receipt_calendar_config): without a chain connection or a typed
[v2] from_block the audit reports a calendar-launch warning and a satellite
holds, rather than trusting a receipt's own stamp.
The 95% pre-launch burn is independent: prelaunch-95-r2 must reach the
configured 51% of eligible permit stake, then starts at the next settlement.
Do not type a burn block merely to bypass the readiness vote.
In order. Each item names the code or load check it comes from; anything the
loader checks fails cascade-trainer --offline / cascade-validator --offline (both only load the config and print), so run those against the
edited chain.toml before shipping it.
- Lock file.
uv.lockincludes the git-pinnedmarketcastextra. Use the committed lock with the extras required by the selected role; validate lock consistency before building a release. Pods need the training dependencies (train) and MarketCast workers also needmarketcast. - Worker image.
deploy/Dockerfileinstalls.[train,hippius,wandb,marketcast]. Rebuild from the intended release and qualify the image, including--router-slowandmkt_finetune. Then runscripts/repin_worker_image.sh <published-tag> chain.toml. It verifies the published package matches this release and atomically updates both training digests,[round] funded_pod_imageand[round] worker_code_fingerprint. SetCASCADE_PYTHON=.venv/bin/pythonif the project dependencies are installed in that environment. A locally built image is not deployable until the registry can serve its pinned digest. - Benchmark sidecar env on whichever box sweeps lane A (the main
validator under DEC-CA-0063, else the trainer's pods): the DEC-CA-0061
rule needs FEV, which is the optional
fevextra (uv sync --project benchmarks --extra fev --extra time;benchmarks/Dockerfileincludes both extras), plus data staged withcascade-benchmark-downloadand[eval] gift_gate_data_dir. - Hardware qualification. The pyramid contract uses explicit
training_steps;ref_throughput_tokens_per_s = 0means unmeasured, not zero work. Measure the 1.23B port on the intended GPU, including accumulation, full-batch gradients, checkpoint I/O and the four public suites. The research run's four-GPU expert parallelism is not a feature of this single-device port. Keep the measured configuration and its image pinned before deployment. - CPU generation tier (DEC-CA-0053, rides
[v2.training] corpus_mode = "shards"): set[generation] vm_image(empty in both TOMLs), verify private corpus-store pre-signed URLs, run one testnet leg end to end. Each leg gets 32 vCPUs for up to 90 minutes (window_seconds = 5400), stopping earlier at 50B points. Provisioning, digest finalization and upload add time outside that generation window. This is operator-side configuration; changing the window does not require other validators to update. Bootstrap imports every supported generator library and prints its version usingscripts/check_generation_runtime.py; retain that manifest when qualifying the production VM image. A dependency import failure must stop provisioning before any miner generator runs. - Main validator identity. Mainnet pins
[netconfig] owner_hotkeyto its configured main validator. Testnet keeps a separate unset identity; set it when enabling validator-sourced benches there. The loader refuses[v2] bench_source = "validator"with a resolved block > 0 unlessowner_hotkeyis set,[v2] king_ladder = true, the block is an era boundary at/after[v2] from_block, andowner_hotkeyequals[manifest] validator_hotkeywhen both are set (cascade/shared/config/v2.pycheck_bench_source). Mainnet shipsbench_source_from_block = "v2", so typing[v2] from_blockwithoutowner_hotkeyfails to load. - Testnet only, optional. Testnet ships
king_ladder = falseand nobench_source(trainer benches). To exercise DEC-CA-0055 / DEC-CA-0063 there, setking_ladder = true,bench_source = "validator",bench_source_from_block = "v2"(or an era boundary after it) and the testnetowner_hotkey(= testnet[manifest] validator_hotkey). - Calibrate the placeholders (all CONSENSUS; each is overlay-settable
except where noted):
[scoring] bench_gate_tolerance(0.01) /bench_gate_min_win_rate(0.5):scripts/bench_seed_noise.pyon ≥ 3 seed replicas of the v2 recipe (DEC-CA-0061).[v2] bench_sweep_hours(2.0) /bench_budget_hours(6.0) (overlay-settable):scripts/bench_timing.pyon the sweep box, with the[eval] bench_workers/bench_pairedyou will run (see Sizing the sweep).[v2] ladder_start_tiers: seed replicas on the private pool (the 0.50 tier must be rare under noise); keep the lowest threshold at[v2] win_margin_end.[lane_b]/[lane_c] full_pay_tiersand[marketcast] full_pay_tiers(overlay-settable): from real win LCBs (testnet receipts'target_lcb/gain_lcb); lowest threshold atgain_margin.[marketcast] near_dup_tol(0.02, overlay-settable) and the liquidity floormin_dollar_volume_24h(release-only): once the recorder shows real volumes.
[v2] from_block. Type an era boundary in the future (the loader refuses a non-boundary; a passed block judges the eras in between under the wrong contract). Trainer and every validator run the v2 release BEFORE it. Testnet is armed at 8190000: confirm it is still ahead of the head before deploying. Leave B and Cfrom_block = 0for the A-only launch: lane A gets the full allocation, with its existing ladder and burn rules. Announce B separately at an era start ≥[v2] from_block; B reserves 25% from its first settlement; A keeps 75% if C is off and 50% if C is already live. Empty B slots burn.[lane_b] acceptance_from_block = "v2"makes individual wins unpaid qualifications: only compatible deltas accepted at a warm start start earning. Testnet leaves it at 0 to preserve settled history. To upgrade a live lane, drain legacy pending merges, announce a future era boundary, and upgrade trainer and validators before that block. The loader validates alignment. Do not disable acceptance while qualified merges are pending; the trainer refuses to merge them through the legacy path. Preserve the lane-B state file: after acceptance activates, a missing state is rebuilt from the full settlement chain back to lane launch (or the saved marker), because accepted-delta identities outlive payout TTLs. Historical rounds fold oldest first; a TTL-sized replay cannot safely reconstruct which expert deltas have already entered the model. Acceptance benchmarks run in a background job. The new init waits while intake and status ticks continue; no new-init leg starts early. Staged results consume pending deltas only when the scheduler adopts that exact era/base, so obsolete jobs and process restarts leave the queue intact.[lane_c] from_block, if typed, must be an era boundary at or after v2 starts. Mainnet C takes its share from A (takes_from = "lane_a") and may launch before B: A/C = 75/25, then A/B/C = 50/25/25 once B launches. A C policy funded from B still requires B to be live; testnet retains that option. The settlement AT a lane's launch boundary still belongs to the preceding era.- MarketCast (only when arming
[marketcast] from_block): startcascade-pool mkt-record(andmkt-record-alphaifalphaassets are listed) at leastsnapshot_days(21) before the block — the loader cannot see the recorder; install themarketcastextra on the main validator; the loader then requires an era boundary ≥[v2]and[lane_c] from_block, the validator bench source in force by then,owner_hotkey, a commit-pinnedbase_ref+base_sha256, and a built[marketcast.data_sources]entry for every class that lists assets (cascade/shared/config/marketcast.pyarming_problems,check_marketcast_arming). See MarketCast.
The transport is plain SSH (cascade/trainer/remote.py), so any
SSH-reachable GPU host looks the same to the trainer.
docker build -f deploy/Dockerfile -t <registry>/cascade-worker:<tag> .
docker push <registry>/cascade-worker:<tag>
docker inspect --format='{{index .RepoDigests 0}}' <registry>/cascade-worker:<tag>Pin the digest in chain.toml, never a mutable tag:
[training]
train_image_digest = "<registry>/cascade-worker@sha256:<digest>"Every pod receives the same digest at launch as CASCADE_TRAIN_IMAGE_DIGEST
(a container cannot introspect its own OCI digest). With the pin set,
cascade-train-worker refuses a run whose runtime doesn't report it, the
trainer probes each pod's worker runtime before dispatch, and cascade-audit
Tier 2 uses the match to decide when a byte-exact checkpoint comparison
applies. Rented pods boot [round] funded_pod_image; pin it as described in
Credentials on payer pods.
train_image_digest is part of contract_digest. Before DEC-CA-0067 is in
force, re-pinning it mid-epoch makes the trainer drop the carried king entry
and every finished leg (_stale_contract) and retrain them (operator-billed),
and validators reject any entry with a different full digest
(era_entry_init_mismatch). Do not re-pin inside a v2 seed epoch until the
switch below is in force.
Once [v2] image_decoupled_from_block is in force for an era, every v2
settlement of that era stamps numerics_contract_digest (the contract minus
train_image_digest) on the manifest and on every entry. Validators gate legs
on that value. An image-only re-pin then keeps the king entry and pointer and
the finished legs. A numerics change still retrains. The pod still refuses a
FINAL whose runtime image differs from the pin.
Deploying this release: the chain.toml values first. Every deployed
chain.toml must carry [v2] eval_n_windows_after = 4000,
[v2] quantile_head_after = true,
[v2] image_decoupled_feature = "v2b-69630f92", and the typed
[scoring] forfeit_from_block = 9166500 / margin_v2_from_block = 9194400.
The feature name hashes eval_n_windows_after and quantile_head_after, so
a file missing either would vote on another name (v2b-1e520694 without
both, v2b-1abeaeb4 without the quantile-head knob) and fork from the fleet.
What the bundle switches (all at the same era start). The image-decoupled
per-leg contract rule, the 4000-window verdict budget, and the quantile-head
verdict: from the switch era a v2 settlement scores each checkpoint's quantile
head directly (the wrapper's forecast_quantiles_batch at the 9-level scoring
grid, the benchmark path's method) instead of drawing 100 copula sample
paths. A validator that offloads private eval to a GPU host must run this
release on that host too — an older worker refuses the quantile request and
the round holds until it is upgraded (never a silent fallback to samples).
Restarting a validator on this release is a vote. With the pins in its
chain.toml, a validator signals v2b-69630f92 in its on-chain note from its
first loop: there is no separate "release, then vote" step. Lock-in follows
at 51 % of stake, and the switch is the era start two eras after the lock-in
boundary's era. The running v2 seed epoch CONTINUES across it (DEC-CA-0067
rule 2 as image-decoupled/3): same seeds, same init, the carried king and
the finished legs pass the decoupled check — no king leg, no init-pointer
rebuild, nothing discarded. A TYPED [v2] image_decoupled_from_block must
give the same notice (an era start at least two eras after the era the node
first loads it in) and must be deployed to every validator and the trainer
before the switch era's pre-train window: a node that loads it later refuses
to start (too little notice) until the switch era begins. So:
- Trainer first. If lock-in happens while the trainer still runs older
code, every upgraded validator rejects every post-switch settlement
(
era_numerics_missing). Restart the trainer on this release (no legs in flight, not inside a validator scoring window) before any validator. - Then validators, any time outside their own scoring window. The
switch does not touch the seed epoch, so there is nothing to time it to.
From the switch the verdict budget goes from
[eval] n_windowstoeval_n_windows_afterwith a new draw: check validator RAM/CPU and the trainer's lane-B self-judge load first. - Announce the release to the external validators only when 1 and 2 hold.
Every validator must run it before the switch era: one still on the old
release rejects every post-switch settlement (
signature_invalid).
What refuses and what keeps running on a file without them:
- Refuse to start:
cascade-validator,cascade-satellite(both signal and tally the vote),cascade-trainer,cascade-audit(they act on it); theirmainsets the loader's vote-pin guard. The validator and the satellite also refuse to start when their own note would trim thev2b-segment, which is what happens with the pins but without the typed[scoring]blocks. - Keep running, ERROR logged: train workers on pods
(
cascade-train-worker, the pod-side snippets), the miner CLI (cascade),cascade-intake(funding),cascade-provisioner,cascade-pool,cascade-generate. - Note headroom (time-limited): the note-budget check judges blocks at
the digit count they reach within ~30 days. With the release chain.toml
(rollover resolved from signals, a locked
cfg-overlay segment) thev2b-segment no longer fits once blocks reach 8 digits, so from head ~9,784,000 a validator on that config refuses to start until v2b's block is typed once decided (or the already-locked rollover is typed). Plan that edit before then.
Note byte budget: the two ways to free room for v2b-. The commitment
field is 128 bytes. The main validator's live note today is 124 B
(cascade-ready:1:rolling-era-king:0:0:forfeit-42b1a486:9165600:9166500:margin-v2-d30eb443:9190800:9194400:prelaunch-95-r2:0:0);
:v2b-69630f92:0:0 adds 17 B, and the locked segment
(:v2b-69630f92:<lock>:<switch>) adds 29 B. The owner picks one of these
(Approve-tier, chain.toml edits on every deployed copy):
-
(a) type the decided forfeit / margin-v2 blocks (required on mainnet):
[scoring] margin_v2_from_block = 9194400 # = the block margin-v2-d30eb443 resolved forfeit_from_block = 9166500 # = the block forfeit-42b1a486 resolved
This is behaviour-identical, because the restored lock-ins become no-ops. Both segments leave the note: 73 B unlocked and 85 B locked on the main validator. The release chain.toml already carries these two lines.
-
(b) retire the UNLOCKED pre-launch burn vote:
[v2] prelaunch_burn = 0.0 # was 0.95; the prelaunch-95-r2 vote never locked
This is behaviour-identical only while that vote is unlocked: the burn shapes pre-v2 settlements only, and only from a decided block, and every future settlement is v2. A validator whose persisted
activation_prelaunch_burn_state.jsonis LOCKED refuses to start withprelaunch_burn = 0; type[v2] prelaunch_burn_from_blockinstead. (b) alone is not enough on mainnet. It frees 20 B, so the note is 121 B unlocked but 133 B once v2b locks, and the startup check (which judges the locked worst case) still refuses. On a repo-config validator (rollover resolved from signals) even the unlocked note does not fit. Use (b) only together with (a), for more headroom: 53 B / 65 B.
At startup a validator logs one activation: STARTUP — this node's note exceeds 128 bytes; segment '<feature>' will NOT be signalled WARNING for each
segment its 128-byte field would drop, so no vote is dropped silently.
Steps (Approve-tier; snapshot each file first):
-
Hand-edit every deployed chain.toml (trainer box
/root/cascade/chain.toml, the main validator's, any satellite / intake copy): add the three[v2]keys and type the two[scoring]blocks. -
Validate THAT exact file with the release loader before restarting anything, as the voting processes will:
PYTHONPATH=<release checkout> <venv>/bin/python -c " from cascade.shared.config.loader import load_chain_config, require_vote_pin from cascade.shared.activation import image_decoupled_feature require_vote_pin(); c = load_chain_config('<deployed chain.toml>') print(image_decoupled_feature(c))"
It must print
v2b-69630f92and raise nothing. -
Only then restart, one service at a time (validator, trainer, audit, intake, provisioner, satellite), and check each log for the startup line and no
image_decoupled_featureERROR.
How it activates: validators on the release signal v2b-<hash> at startup. At
the first settlement boundary where [activation] threshold of eligible stake
has signalled, the vote locks in. The switch is the first era start after that
boundary. Trainer and validators resolve the same block. A typed
[v2] image_decoupled_from_block is the owner override. The same block also
switches the v2 verdict budget to [v2] eval_n_windows_after.
Re-pin runbook, once status/chain.json (activation.image_decoupled) shows
the lock-in:
- Wait until the switch era has started (head block >= the activation block).
- Append the CURRENT
[v2.training] train_image_digest, exactly as typed, to[v2] prior_train_image_digestsin the deployed chain.toml. The trainer uses it to prove that a pre-repin leg differs only in its image. A leg it cannot prove is retrained and logged at ERROR. - Re-pin
[v2.training] train_image_digestand[round] funded_pod_imageto the new release (scripts/repin_worker_image.sh). Pushchain.deployed.tomland spot-check one pod per stage. - Restart the trainer, preferably with no legs in flight. A leg dispatched
before the re-pin is re-attached after the restart: under the decoupled
rule the restore check and the receipt check accept a digest that is
provably the era contract with the old image (
_stale_contract,leg_pins.receipt_contract_ok). A leg with other numerics is still requeued. - Verify that the next settlement keeps the king (
numerics provenat INFO in trainer.log) and that the validators accept it.
Instead of the hand repin above, the trainer can switch images by itself at the
switch era. Set these in the deployed chain.toml [v2], all together:
train_image_digest_after = "sha256:34375845b2c4412ccd8462a6e9ca5ad78837567115bc7f9b56320458ba991409"
funded_pod_image_after = "ghcr.io/tensorlink-ai/cascade-worker:worker-v2.0.0-rc16@sha256:34375845b2c4412ccd8462a6e9ca5ad78837567115bc7f9b56320458ba991409"
worker_code_fingerprint_after = "5fe6ed7c0f0d5305b6101e589aab50b2ca5a76573cbc7afb3f44bc7354ce8fd0"
# only while [round] funded_sku_wall_measured_image is set:
# funded_sku_wall_measured_image_after = "sha256:<the image the walls were measured on>"These values are rc16. The repo chain.toml ships them empty.
Owner rule (enforced). Set an image as *_after only after a
bit-identical qualification against the current image: weights and optimizer
state, plus crash/resume, at every world size the trainer can rent for a v2
leg. Those are the GPU counts of [round] funded_pod_skus / funded_pod_sku
(mainnet 8xH200, 8xH100, 4xH200, 4xH100 ⇒ worlds 4 and 8; world 1 only with
a one-GPU SKU). Adding a SKU with a new GPU count needs that size qualified.
Record the qualification in OPSLOG first. The trainer enforces it: without a valid
qualifications/<from12>-to-<to12>.json for exactly this pair it refuses
every post-switch rent and dispatch on the after image (ERROR every tick plus
an image_after_unqualified operator alert; legs wait as unburned operator
faults, never on the old image). Make the record from the harness outputs:
# on the qualification pod: A = checkout from the FROM image, B = from the TO image
python scripts/qualify_pyramid_bitwise.py --ref /root/qual-a --cand /root/qual-b \
--worlds 4,8 --chain-toml /root/cascade/chain.deployed.toml --out /root/qual-out
# copy /root/qual-out back (ckpt/ dirs + summary.json), then on the box:
python3 scripts/write_image_qualification.py \
--from-image sha256:<current> --to-image sha256:<after> \
--from-fingerprint <current worker_code_fingerprint> \
--to-fingerprint <worker_code_fingerprint_after> \
--run-id <run id> --qual-out /root/qual-out --manifests-dir /root/qual-manifests \
--chain-toml /root/cascade/chain.toml # the DEPLOYED file: required worlds from its SKUsCommit the record it writes and deploy it with the trainer code. It writes
nothing when any world, the resume check or any tensor is missing or differs.
Emergency only: [v2] image_after_unqualified_override = true uses the after
pins unqualified (ERROR every tick, image_after_override alert).
What changes, and when:
- Nothing changes until the vote resolves the switch (or a typed
image_decoupled_from_block). Every pin, file and dispatch stays byte-identical. - Legs of eras starting at or after the switch get the after pins: rented pod
image, worker-code check, wall gate, the
[v2.training]image in the leg's contract, and a deployed chain.toml with those pins substituted (chain.deployed.after.toml). - Legs of earlier eras keep the current pins, and in-flight legs keep their image.
- Before every dispatch, the worker-code fingerprint of each lane and pod is checked against the leg's era pin. A host on the other image never takes the leg.
- The pre-switch pin joins the numerics proof automatically, so the carried
king is kept without editing
prior_train_image_digests. - No trainer restart is needed at the switch.
Lanes:
- The lane pool hands each leg a lane on its era's image.
- The provisioner health gate accepts either release.
- A lane re-bootstrapped onto rc16 is picked up within 300 s (the fingerprint cache TTL), and is re-measured before any refusal.
- A refusal never trips the lane breaker.
Lane and pod behaviour once armed:
- Lane matching acts only on legs of eras under the RESOLVED switch. A pre-switch leg, and every leg while the vote is unresolved, is handed lanes and dispatched exactly as without the repin (no probe; the worker's own FINAL image check still guards it).
- A post-switch leg takes only a lane whose worker code AND runtime image
identity (
CASCADE_TRAIN_IMAGE_DIGEST, what the worker checks) are the after release; other lanes stay idle for the other era's legs. A lane seen on another release is re-probed at most once a minute. - An ssh failure while probing a lane makes it "unknown": the leg waits and retries; it is never told "no lane on this image". An ssh failure at dispatch is a transport failure (the pod is drained like any unreachable pod), not an image mismatch.
- Never the wrong image (owner rule 2026-11-02), post-switch legs only (a
pre-switch leg is handled exactly as in the release before): a pod or lane
that cannot run the leg's image is released at once and the leg retries
unburned on the same billing. Its host and executor are marked bad for
24 h (a lane leaves the rotation) ONLY when the trainer itself measured the
pod's worker code as another release; a mismatch of the image env we
injected, our deployed config, or a probe that could not run is never
blamed on the host. A worker's own FINAL refusal counts only as exit 2
with the worker's stdout sentinel as its last line
(
contract.IMAGE_REFUSAL_SENTINEL, printed before any miner code runs; workers that predate it never trigger this path) — never from free text. At most 2 such pods per leg (hotkey, role, era) per hour, counted in<work-root>/image_refusals.jsonacross restarts; then the leg waits (no_capacity, requeued), so a miner is never billed repeatedly for our image skew. - The runtime image probe reads what the worker will see: the host's
static_env(sourced into the worker's session), elseprintenvin a plain ssh session (the worker runs in one,remote.build_remote_command). A probe whose remote command fails (python, OOM, disk) is "unknown". - MarketCast legs of a post-switch era carry the era's
--contract-block, so they get the image check and the after-image config. Direct-MoE gangs (cascade/direct) rent their own signed, qualified pod profiles and are not touched by the*_afterpins. - The bench-only king pod is rented under its era's contract.
- A typed
[v2] image_decoupled_from_blockmust be an era start. While the DEC-CA-0045 rollover is still decided on chain it is checked on the post-rollover grid ([activation] epoch_blocks_after × era_settlements).
Gaps:
- The provisioner still pins
CASCADE_TRAIN_IMAGE_DIGESTon its pods from the base[training]pin (pre-existing). Lanes for v2 legs should be rented pods or hand-managed hosts.toml lanes. - An era-kept king pod is reused within one seed epoch. If the king must retrain after the switch, the hook refuses the old pod; retire it.
- Continuation for post-switch legs only, in the same step, is a follow-up:
[round] ops_continuation_*is global.
continuation_protocol is excluded from the numerics digest along with the
image pin. Once the DEC-CA-0067 switch is in force, turning on mid-run backup
checkpoints keeps the carried king and the finished legs. The trainer tries both
protocol values ("" and cascade-continuation/1) automatically, so there is no
list to maintain for this term. Before the switch it is a full contract change:
legs are retrained and validators reject them. Do not arm it earlier.
- Wait until the switch era has started.
- In the deployed chain.toml:
[v2.training] continuation_protocol = "cascade-continuation/1";[round] ops_continuation_image_digest= the qualified worker image's immutable digest (it must equal the runningCASCADE_TRAIN_IMAGE_DIGEST, or the worker refuses);[round] ops_continuation_interval_seconds = 1800;[round] ops_continuation_max_bytes = 34359738368(32 GiB; an rc16 leg with its EMA snapshot is about 24 GB, and the default is 16 GiB).
- If the image changes in the same step, append the old pin to
[v2] prior_train_image_digestsas in the re-pin runbook. - Point the box-side snapshot mirror at the large volume. The orchestrator
mirrors to
<--work-root>/_continuations(cascade/trainer/runner_rolling.py_dispatcher). There is no separate flag, so either run the trainer with--work-rooton the large volume, or replace_train_work/_continuationswith a symlink to it while no leg is in flight. A dedicated--continuation-rootis a follow-up. The pod side writes to<host workdir>/_continuations/<job>(continuation_transport.py); that is set per host in hosts.toml. - Restart with no legs in flight. Detached dispatch is required, and the dispatcher refuses continuation unless a training contract declares the protocol.
The GPU type is chosen per leg: the trainer rents the cheapest fitting
executor across [round] funded_pod_skus under the price caps
(funded_sku_per_leg), with [training] expected_gpu = "". Equal compute
comes from the fixed series-point budget, not from matching silicon.
A miner may choose the type itself (cascade fund --gpu …, DEC-CA-0060:
gpu, gpu_allow_fallback, gpu_max_price_per_hour on its queue entry).
A re-fund that changes that choice reaches a lane-A leg already waiting
for capacity: every poll re-reads the in-flight entry (same ref and
attempt; not a new ref queued behind the flight) and, on a change, logs
funded leg X: miner changed GPU choice A → B and switches the next rent to
the new type, cap and latest safe start. A type that can no longer land
before the leg's target settlement is never rented (the doomed-leg check
requeues the leg unburned). An unreadable queue keeps the current choice.
King legs, default-path legs and lane B / marketcast legs (explicit
choices) are never re-steered; a pod already rented is never touched.
Pinning one SKU (expected_gpu = "<exact torch.cuda.get_device_name string>")
is a LOCKED contract term — a coordinated validator upgrade — and buys
byte-exact re-derivation of checkpoints on that SKU + image.
Rolling submit/fund supports an ordered Lium/Shadeform/Vast.ai/Runpod key list. Upgrade intake
and trainer together, then confirm status/v2.json advertises
compute.rolling_providers. Miners use --provider-key-env PROVIDER=ENV_NAME;
legacy LIUM_API_KEY requests remain valid. cascade credentials updates the
list without queuing work. The signed bundle binds provider names, order and
key digests; keys never enter public queues or rental ledgers.
Read-only capacity selection preserves GPU/hourly/estimated-leg caps. Once a
create is attempted, no second provider is tried on an uncertain result.
Existing allocations are discovered first, including via retained credentials
from an earlier list. Each rental writes a provider and payer_credential_id
into its ledger before use. Credential snapshots live under the payer vault's
leases/ directory, with the same sealing key, permissions and TTL. Preserve
that directory during backups/restarts until all rentals are confirmed gone.
Opaque Shadeform UUIDs are resolved alongside instance names during orphan
cleanup. An inaccessible account blocks new spending; surface it for repair.
A successful authentication check does not establish available credit.
Vast.ai follows the official vast-cli request shapes: offers by gpu_name
(spaces, e.g. RTX 4090), on-demand, verified, priced by dph_total under the
hourly and per-leg caps; creates with client_id = "me"; the paged
/api/v1/instances/ list. An explicit refused create is capacity; a lost,
malformed or contradictory response, timeout or 5xx keeps the create unresolved.
Cleanup requires a successful listing with valid rows, counts and complete
pagination; an API error or partial listing cannot establish that a pod is gone.
Qualify it on a real account before relying on it: GPU
name strings for A6000/H100, listing lag after create, and that deletion stops
billing. Runpod is accepted by rolling intake and cleanup; its on-demand
adapter refuses creates with an hourly or estimated-leg cap. Provider selection
skips that capped account and may choose another listed provider before any
create. With no compatible account, status reports price_cap_unavailable.
Only an explicitly chosen GPU with --max-price 0 uses Runpod's uncapped path.
Do not interpret this as a hard account or total-dollar budget.
Lium documents key budgets configured by the human. Shadeform has a native
per-instance auto-delete spend threshold: applicable funded_max_leg_cost_usd
is sent to the create API, rounded down to cents. It covers that instance only.
Vast and Runpod hard per-key budgets remain unconfirmed; permissions, alerts
and balances are not substitutes. Discovery publishes these distinctions in
compute.spending_controls, without claiming to verify any key's allowance.
The trainer rents miner-funded pods on Lium, Shadeform, Vast.ai or Runpod
(cascade/provision/funded.py, cascade/trainer/loop.py):
-
Payer pods: one per funded challenger leg, billed to the miner's own provider key from the sealed vault. Torn down after harvest and any required trainer-sourced benchmarks. Validator-sourced A benchmarks run separately.
-
King training: under v2
king_ladder, normally billed to the king's own key; earlier rules and failed-key/operator recovery can use the operator path ([round] funded_king_rent). Seed epochs can reuse a trained king across eras. Trainer-source benchmark verification can retain an operator king pod; do not assume a new operator-funded king rental every era. -
Operator-billed legs: only with
[round] funded_operator_fallback = true(shippedfalse). Then a leg still waiting for marketplace capacity takes a free operator lane fromhosts.toml, and a leg whose payer key has expired runs on an operator lane or, with none on file, on a pod the trainer rents on the operator's key. With the flag off, payer legs never run on operator hardware, and a leg whose key has expired is released asauth, unburned.[round] operator_lanes_operator_billed_only = true(shippedfalse) narrows the lanes to operator-billed work: make-goods / expired-key entries (operator_billed) and king legs. A miner-funded leg with no marketplace capacity then requeuesno_capacity, unburned, instead of taking a lane. An operator-billed entry requeued for an operator-side fault backs off fromoperator_billed_retry_seconds(default 60), doubling per consecutive fault up to 30 min (0 = the infra base of 120 s). -
Every operator-billed miner leg needs the owner's approval (2026-11-02). No path bills a miner leg to the operator on its own. This covers an expired payer key,
funded_operator_fallback, operator lanes,ops_paid_fallback_after_secondsand a make-good after our fault. It also covers a retrain after a finished run was lost on our side, even on the miner's key, because the miner already paid once. Such a leg parks asawaiting_operator_approval. It is unburned, never re-dispatched and never TTL-expired, and it raises oneoperator_leg_awaiting_approvalalert. That alert is critical when it carries the miner's lost cost. The owner acts per leg:cascade-trainer list-operator-legs # pending + paused + rent halt cascade-trainer approve-operator-leg <hotkey> [--max-cost 120] [--kind KIND] cascade-trainer deny-operator-leg <hotkey> cascade-trainer resume-operator-leg <hotkey> # a paused entry, cause fixed cascade-trainer rent-status # the box-health rent halt cascade-trainer resume-rents --reason "disk freed" # owner override of the halt
Run these from the deployment directory (
--work-rootdefaults to./_train_work).approvewith no pending request pre-approves the hotkey's next operator-billed leg;--kindnarrows it.An approval lasts for one leg. It is bound to the entry's ref and funding and to the request's kind, and is used only when that leg's pod is actually up (or its lane dispatched). Only then is the operator spend booked. A sold-out market, a lemon or a rent error keeps the approval, and a restart re-attach of the same attempt passes again. A retry after the leg failed needs a new approval. A re-fund voids it.
A leg that already holds its operator pod is adopted with no approval, preflight or spend booking. That covers a ledgered
…-op-0row on a restart, and an operator-billed in-flight leg re-attached on a lane. If that pod is gone, the fresh rent is gated.A denial sends the leg back to the miner's own key. A leg with no key fails unburned, with the reason "re-fund needed". A miner-funded leg with a working key leaves its market wait for operator capacity only when an approval or pre-approval is on file. The king is not gated.
-
Operator-fault cap. Only consecutive real operator faults of the same cause count (
error_class "operator"/"operator_kept_pod"). These never count: capacity waits,no_capacity, operator-rent refusals, a VM that never booted, a failure with no recorded verdict, and infra on an operator pod (alloperator_transient, backed off). A different real cause restarts the count at 1.At
[round] operator_fault_cap(3) the entry ispaused: not failed, not burned, oneentry_pausedalert. It resumes by itself afteroperator_pause_resume_seconds(21600 = 6 h; 0 = manual only) or viaresume-operator-leg. The count is mirrored in every phase event, so a rewrite by older code keeps it, and the pause lives in the status itself. -
Money gates before any rent. These run before every rent:
- The box-health halt. See Disk space and harvest safety.
- A pre-rent canary. A write, fsync, rename and delete in the harvest dir.
- A disk reservation. Free space on the harvest filesystem must be at
least
expected_checkpoint_gb(15) × (live legs holding a pod + 1) +min_free_disk_gb, measured after eviction. A canary failure trips the halt. A reservation shortfall refuses only that rent and never halts. - For an approved operator rent:
- the breaker is closed;
- the projected cost is within
ops_operator_leg_cap_usdand the daily cap (operator_daily_spend_cap_usd, default 200, UTC day). The projection uses the cheapest current offer's $/h for the round SKU (else theoperator_max_price_per_hourceiling) × the expected wall. The spend booked at launch is the landed $/h × the wall, so the $200 default admits about two 8xH100 legs a day at ~$12–15/h.list-operator-legsshows booked vs projected for today; - the operator Lium balance is at least projected cost ×
operator_cost_safety_factor(1.25) +operator_balance_reserve_usd(50, which keeps the validator's eval pods alive). An unknown balance or a shortfall refuses that leg only, never the breaker.
- An optional payer-balance check (
payer_projected_rate_per_hour, 0 = off).
A blocked leg requeues unburned as
operator_disk/operator_blocked. It backs off and is never counted against the cap. One alert is raised per block. -
Operator-account circuit breaker. Only a real 402 / BALANCE_REQUIRED from an operator launch opens
_train_work/operator_breaker.json. While it is open, no challenger operator rent runs foroperator_breaker_cooldown_seconds(3600). The king's JIT pod and king backups are never refused by it; only their own 402 trips it. It raises oneoperator_breaker_openalert.
The carried king's first v2 training. The identity carried from v1 is
its hotkey/generator, not a Toto2 checkpoint relabelled as a v2 model. Its
selected immutable generator must first pass v2 launch qualification with
exactly 32 channels. CPU generation is operator-paid. On mainnet, the king's
GPU leg first uses its deposited provider credentials; it does not need a new
challenger funding request. If those credentials are absent, rejected or out
of credit, the operator's account pays for the king GPU. This recovery is
independent of funded_operator_fallback, which controls challenger fallback.
The operator CPU request retains king priority in either funding path.
Under the existing reward policy, a key-caused operator fallback records
funded_by="operator" and can cap the carried king at 1% of A rather than its
normal at-most-5% of total emissions. Provider/market/operator recovery records
operator_fault and does not open a key lapse. There is no special free first
GPU or launch grace exemption. If the operator also has no credit/capacity,
the king retries and dependent settlements wait; challengers are not charged
for its training. The default operator account is Lium. Additional operator
providers require the capped operator pool and its explicit per-leg/daily
limits. Verify that account is funded before admitting the launch cohort.
Testnet's default king_ladder=false uses operator king funding and does not
exercise the mainnet self-funding/penalty path. Its default streaming corpus
also does not qualify the CPU shard tier. A successful streaming testnet run
alone is insufficient evidence for either launch behavior.
Every one of these pods is isolated and credential-free (details:
Credentials on payer pods). Public Hub
generators and warm-start inits pull anonymously; private (vault)
submissions are staged onto the pod as a ZIP. Pod names are
deployment-scoped (cascade-n<netuid>-<round>-funded-<slug>) so co-hosted
deployments on a shared payer account never reap each other, and the trainer
reconciles and reaps them through its write-ahead ledger
(<work_root>/funded_pods.json, swept at trainer startup; pods of legs still
in flight and the current eras' king pods are kept and re-attached). The
provisioner's orphan reaper never touches them.
The trainer needs LIUM_API_KEY (operator account) and a hosts.toml
whose first final/any entry is the pod profile every rented pod mirrors
(image paths, the orchestrator SSH keypair <key_path> + <key_path>.pub).
A profile-only entry (profile_only = true, or a loopback address) is
enough; it is never a dispatch target.
Hand-rented GPU hosts serve the operator-billed legs above. Point the trainer at a hosts file; it is re-read on every use, so a lane added later is picked up without a restart:
cascade-trainer --remote-hosts hosts.toml # + wallet/chain flags[[host]]
name = "a6000-oslo-g0"
host = "203.0.113.10"
user = "root"
key_path = "~/.ssh/trainer_orchestrator" # the PRIVATE key
remote_python = "/root/cascade/.venv/bin/python"
workdir = "/root/cascade" # matches the image WORKDIR
cuda_device = "0"
isolated = trueTwo GPUs on one box → one entry per card (cuda_device = "0" / "1"); see
scripts/remote_hosts.example.toml. A lane joins the pool only after a gate:
SSH answers, the pod runs this release's worker image, and the host clears the
per-SKU bench floor. A lane whose SSH is refused is quarantined.
An entry with stage = "init" is an init-only lane: the trainer uses it
solely to build the seed epoch's random-init checkpoint (the DEC-CA-0063
pointer the main validator benches) when the epoch has no operator king pod
and no idle operator lane — never for a leg, never counted as capacity, so no
miner code runs there and isolated is not needed. Rent the cheapest pod that
runs the worker image, add the entry (the trainer pushes its deployed
chain.toml before the CPU-only build, which takes minutes), and remove the
entry once the trainer logs random-init checkpoint built on init lane. Mind
the timing: the pointer first appears in the trainer's bench report of the next
settlement, and the king check that reads it should fall on the first or second
settlement of an era so the fired promotion still has its full notice.
Keep isolated = true on every lane: miner code runs there. forward_env
(forwarding HIPPIUS_* credentials so the pod pushes its own checkpoint)
remains only as a deliberate exception, and the trainer logs a WARNING at
startup for every non-isolated host that forwards secrets.
Launching a lane: run the image with SSH exposed and your orchestrator public
key injected, e.g. on a base GPU pod with docker + nvidia-container-toolkit:
docker run -d --gpus all -p 22:22 \
-e SSH_PUBKEY="ssh-ed25519 AAAA... trainer-orchestrator" \
-e CASCADE_TRAIN_IMAGE_DIGEST="<registry>/cascade-worker@sha256:<digest>" \
<registry>/cascade-worker@sha256:<digest>Pods run miner-controlled generator code. Configure the sandbox in
chain.toml [generator]:
[generator]
# EITHER: subprocess sandbox that REFUSES to run when the host cannot provide
# a network namespace. Needs unprivileged user namespaces on the pod
# (`unshare --user --map-root-user --net true` must succeed).
sandbox_strict = true
# OR: kernel-enforced container sandbox — docker/podman with --network=none,
# --cap-drop=ALL, no-new-privileges, read-only rootfs, tmpfs workdir and
# memory/pids/cpu limits. Needs a container runtime and docker.sock access.
sandbox_mode = "container"
sandbox_image = "<registry>/cascade-worker@sha256:<digest>"
sandbox_python = "/root/cascade/.venv/bin/python"Never disable the checked-in strict sandbox in production (sandbox_mode = "subprocess", sandbox_strict = false): on a host without unprivileged user
namespaces it silently leaves only the in-process socket guard between miner
code and the network. The same rule holds on the orchestrator, which holds
the private eval pool and the trainer wallet: generator code never runs there
without sandbox_mode = "container" or sandbox_strict = true.
Two layers enforce this (cascade/shared/sandbox_policy.py). A load
check refuses any config knob that would run generator code on a weak
sandbox (today only the retired [round] dedup_probe_series > 0, which
nothing reads). A runtime guard: cascade-trainer and
cascade-validator mark themselves at start as holding secrets, and every
generator entry point (trainer/sandbox.py, trainer/corpus.py) then
refuses in-process generator execution always, and a sandbox child unless
the sandbox is hardened (SandboxPolicyError, an operator fault, never booked
against the miner). Payer / worker pods, the CPU generation tier and miner
tools never mark themselves, so nothing changes there.
No-egress lanes. A credential-free worker needs outbound access only to the generator / init sources it pulls from (the Hippius registry and S3) plus inbound SSH from the orchestrator. Deny everything else at the pod boundary:
for host in registry.hippius.com s3.hippius.com; do
for ip in $(getent ahostsv4 "$host" | awk '{print $1}' | sort -u); do
iptables -A OUTPUT -d "$ip" -p tcp --dport 443 -j ACCEPT
done
done
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # SSH replies
iptables -P OUTPUT DROPProviders with security groups: express the same policy there — 22/tcp inbound from the orchestrator only, outbound only to the storage endpoints.
cascade-provisioner (cascade/provision/, config in
scripts/provision.example.toml / deploy/provision.*.toml, unit
deploy/cascade-provisioner.service) runs beside the trainer. It never calls
the trainer; the two meet only through files and stores. It holds the
marketplace API keys, read credentials for the manifest store and the
orchestrator SSH keypair — never the wallet. Each cycle it:
- Eval stage. When a new manifest appears, rents one GPU pod for the
validator's heavy evals and publishes it to a separate
eval_hosts_pathfile the validator re-reads per eval (never the trainer's hosts file). It tears the pod down when the settlement's receipt publishes, a newer manifest supersedes it, or the TTL fires. No capacity or a failed health gate means the validator evaluates locally — degraded, never lost. - Hosts file. Publishes the trainer's hosts file atomically: the
static_hostsfragment (re-read every publish; a broken edit keeps the last good copy) plus any[[host]]entries it does not own, preserved verbatim. Ownership is by address. Do not name a hand-rented pod in the provisioner's own shapes (cascade-<provider>-<8 hex>-g<n>lanes, orcascade-<digits>-(heat|final|eval)…on the provider), or it is treated as the provisioner's and dropped or reaped. - Orphan reconcile. Terminates live provider pods tagged with its own name shape that its ledger does not own, so a crash between launch and ledger-save never leaks a billing box.
- TTL backstop. Every pod it rents dies one epoch after rent no matter
what;
max_spend_per_roundcaps the worst case before anything is rented. - Adopted-pod re-gate. After a restart, pods adopted from the ledger are
re-checked by the health gate before reuse. The gate's
fresh_bootcheck also fails a recycled container (one that already pinsCASCADE_TRAIN_IMAGE_DIGESTor has a non-empty_train_work). - Grid activation. Reads the DEC-CA-0045 on-chain activation so its epoch grid switches at the same block as the trainer and validators.
Its config accepts and ignores the tables of the retired per-round fleet
([provisioner.heat], [provisioner.final]); the trainer rents the king and
every leg itself. On funded pods the provisioner must never: rent them on
the operator key (provision/funded.py has no such path — keep it that way);
forward shared credentials (HIPPIUS_*, HF_TOKEN, WANDB_*); auto-retry a
rent in place (every attempt spends the miner's budget — retry cadence lives
on queue requeues, should_recover + attempt caps); or touch any
cascade-n<netuid>-… pod.
Design rationale: DEC-CA-0036 (funded legs, intake, vault, direct
submissions) and DEC-CA-0043 (rolling intake). The miner side is documented
once, in MINER.md §5 (submitting and funding) and
§8 (what re-queues, what spends the
submission); keep that table in sync with classify_funded_worker_failure
and the queue's terminal states when either changes.
| piece | where |
|---|---|
| Payer-key vault (TTL, sealed at rest, hydrate-for-teardown) | cascade/funding/vault.py |
| Rent-fault taxonomy (auth / rate / capacity / infra, no-burn rules) | cascade/funding/faults.py |
Funded queue (reveal-block seniority, one live entry per hotkey, in_flight state) |
cascade/funding/queue.py |
Intake service (cascade-intake, signed X-Lium-Api-Key POST, direct submissions) |
cascade/funding/intake.py, main.py, store.py |
| Rolling scheduler (admission, per-leg GPU choice, settlements, era king) | cascade/trainer/rolling.py |
Per-payer rentals, verified teardown, write-ahead ledger funded_pods.json, payer-key reap |
cascade/provision/funded.py, cascade/trainer/loop.py |
| Duplicate registry (earliest commit keeps; king + published champions included) | cascade/trainer/dedup_registry.py |
| Checkpoint guard (a checkpoint is data, never code) | cascade/eval/checkpoint_guard.py |
Transparency: roster funded/round-<id>.json + funded/latest.json, cascade queue, tier-0 funded-roster audit check |
trainer/rolling.py, miner/cli.py, audit/checks.py |
Miner commands (cascade submit, cascade fund / --withdraw, cascade queue, cascade iterate, cascade direct; --json contract) |
cascade/miner/{cli,iterate,direct,output}.py |
Validator settlement recovery uses <state_db>.settlement.json as an undo
journal across champion, Cascade, lane-B and lane-C files. Keep these files
together on durable local storage. An interrupted fold is restored before
replay; a committed fold retains its pending receipt/weight follow-up. Disk
write failures hold progress. Do not remove a prepared journal to bypass an
error. The validator entrypoint holds <state_db>.lock so two processes cannot
write the same bundle. A fresh bundle still needs the historical evidence
needed to reconstruct consensus state; it is not a substitute for backups.
After a restart, persisted open B/C legs count as operator-side losses. A
verified pending upload can still be reused; a required retrain follows the
operator_billed rules below. An expired lane-B leg keeps its hotkey busy
until its worker returns; scientific expiry does not terminate a GPU process.
Required lane-B public suites must have min_public_items usable pairs for
both CRPS and MASE. Different known data revisions, one missing revision pin,
missing suites or insufficient required evidence hold trainer settlement or
merge preparation. Historical evidence with neither revision pin remains
readable; it does not prove the data versions were identical.
Once lane B, lane C or validator-sourced lane-A benchmarks are active, every boundary with a valid era king can publish a king-only settlement. Lane A needs no new challenger for B/C results, benchmark acknowledgements, confirmations and reward clocks to advance. The existing signed manifest format and validator checks apply; an unavailable king still holds publication.
[round] funded_field_cap now also caps shared admission of miner-funded
A/B/C jobs preparing data, waiting for capacity or training. Waiting jobs
enter FIFO at this shared gate; per-lane seniority and target caps still
apply. The king has reserved admission and gets first GPU acquisition.
funded_field_cap is overlay-settable. CPU provisioning has a separate
[generation] max_parallel_jobs cap (2 in both TOMLs), so a large GPU queue
cannot launch unbounded generation VMs. Admission waits do not rent a GPU.
These are per-trainer-process controls; run one authoritative scheduler.
Without it, a shard-mode corpus is built only inside an admitted leg, so
entries waiting for a GPU, the coldkey rule or the latest-safe-start deadline
leave the CPU tier idle. With pregen_enabled = true (default false) the
scheduler starts, each tick and in reveal order, the CPU build of every
queued entry whose funded ref is revealed and whose corpus is missing for the
seeds its leg would train under (the next era's once that era is prepared).
No GPU admission, deadline or coldkey gate applies; the build is
operator-billed. Corpora are keyed by base seed and ref, so they are reused by
the leg and every retry within the seed epoch; a new seed epoch stops pregen
for the old seed (running builds finish).
pregen_max_parallel(default 4): at most this many pregen builds hold CPU slots. They share themax_parallel_jobsgate but only fill idle slots, and their slots never count against a real leg, so a leg never waits behind pregen (up to2 x max_parallel_jobsVMs can run while pregen drains). A pregen still waiting for a slot steps aside when its own leg starts.pregen_max_usd_per_epoch(default 50.0): estimated spend cap per base seed. Each start is booked atmax_price_per_hour x (max_window_seconds + 2 h overhead)in<work_root>/pregen_spend.json(refunded if nothing was rented); no new pregen starts once exhausted (logged once).
A pregen failure is logged and retried with back-off (30 min, doubling, max
6 h); it never fails, burns or bills the queue entry or touches
operator_faults. A generator fault is not retried; the entry's own leg
decides it. The per-hotkey CPU rent cap counts pregen rents too.
GPU selection and launch stay serialized until the executor is claimed.
Once its identity is known, boot, SSH and runtime checks release that
selection lock, allowing other jobs to acquire different GPUs. Unknown
executor identities retain the lock conservatively. All three lanes use
their own queue and actual leg budget throughout rent and harvest; lane B
uses its leg_hours in the orchestrator as well as the worker.
An operator/provider failure after allocation, or a completed leg lost to
an era/routing change on our side, marks that entry operator_billed.
The flag persists across requeues and restarts and is visible in queue
JSON, with operator_faults counting each loss. Without operator capacity,
every allocation that billed the miner and delivered nothing is a loss of its
own — a lemon that never became ready, a pod released for a stale image or a
slow bench — and a replacement is rented within the same attempt only while
the losses stay under the cap, so a systemic fault bills the miner at most
OPERATOR_FAULT_RETRIES allocations in total before the pause (not three
pods per attempt). A retry first reuses any
verified pending upload. When an operator lane or the capped operator rental
policy below is configured, the retrain runs there (B/C keep their role, base
and recipe/selection). Otherwise the flag is cleared and the leg rents a new pod
on the payer's key; a pod whose release is unconfirmed still blocks that rent.
After OPERATOR_FAULT_RETRIES (3) losses the entry pauses with an operator
failure instead of billing the miner again. Fix the cause, then (with the
trainer stopped) remove operator_faults from that queue entry to resume it.
That is separate from the operator_fault_cap pause above, which
resume-operator-leg clears.
The flag is a recovery billing restriction,
not a refund of already consumed GPU time. No-capacity failures before an
allocation remain ordinary retries without an attempt burn.
The checkpoint volume once filled to 100%. The validator eval cache held 401 GB of never-evicted checkpoints, and every finished leg then failed its harvest with ENOSPC. Each failed leg was re-rented on the MINER's key and retrained, failing again the same way: about $640 and 5 h before anyone noticed. Now:
-
The box-health breaker (
cascade/trainer/box_health.py,_train_work/box_health.json). Every our-side fault on a leg (local disk/IO, a harvest we could not finish, our storage) is recorded, with one critical alert. Any local-disk fault, or the same class on ≥ 2 legs withinbox_fault_window_seconds(7200), halts every new GPU rent: miner keys and operator, new legs and retries. For a disk fault the king's JIT pod is halted too, loudly. Legs already training keep running. Waiting legs requeue unburned (operator_disk) and never count toward any cap.Only a real disk error or a failed canary is a disk halt; a plain reservation shortfall refuses that one rent and never halts. The halt is re-checked right before every launch, and in-process it holds even if its own state write failed. It clears itself when its health check passes:
- A disk or harvest halt. A write + fsync + rename + delete canary in the harvest dir passes, and free space holds every live pod-holding leg's checkpoint plus one more and the reserve.
- A storage halt (our registry push lost a checkpoint). The Hippius Hub
registry takes a push of one fixed tiny file to one fixed repo
(
<namespace>/box-health-canary, the same digest every time). The probe is Hub-only and never goes through the HF / R2 fallback. With no hub configured, onlyresume-rentsclears it.
resume-rentsfrom another process clears a persisted halt everywhere. Only a halt whose own state write failed is latched in-process.A disk halt also holds the king and king backups; a harvest or storage halt never does. A halted king rent waits at the poll cadence with no provider call; it is never counted as a king-leg failure and never rotates or quarantines a host. Disk errors outside the leg paths also halt: the pod ledger, pending-upload records and the corpus write. A miner's unreachable generator artifact is never our storage fault. A kept pod's re-harvest is never held by the halt; it only needs room for its own checkpoint.
The owner can also clear it with
resume-rents, which takes a reason.rent-status/list-operator-legsshow the halt. -
No blind retrain. A leg that an our-side fault hit after it trained re-harvests its kept pod; it is never retrained. Before any rent, the kept pod's liveness is asked through the payer's provider. A blip is retried and never reads as "gone". The pod is kept while the keep costs less than
harvest_keep_cost_fraction(0.5) of the run it protects (keep hours < 0.5 × trained hours) and never pastharvest_keep_max_seconds(10800), and is re-attached every 60 s. The pod → R2 push is always tried before a release when R2 is configured (a critical alert says so when it is not). If the finished run is gone, the retrain is an OPERATOR make-good (make_good, owner approval, the miner's loss priced in the request) — never on the miner's key again; without operator capacity it waitsoperator_blocked. One critical alert prices what the miner already lost. A pod that has run training is torn down only after a verified harvest or when the run is proven failed. Every miner-billed keep and loss raises a critical alert so the owner can make it good. -
One space manager (
cascade/shared/disk_space.py) covers the validator eval cache,_bench_ckpts,_funded_ckpt_guard, settled_funded_harvestcopies and_funded_harvest.box-*(pending check included). Each source has a byte budget and each filesystem a free-space floor. Reclaimable items go oldest-first, by the newest mtime in their tree. These are never evicted:- a checkpoint referenced by the validator state,
cascade_state.jsonorwarm_start_init.json(the king included); - the manifest being settled (
<cache>/.pinned.json); - a harvest whose push is not confirmed (
*.pending.json) or that is being written; - anything touched within the grace.
The trainer and the validator serialise on
<cache>/.space.lock. When the protection state is unreadable, nothing is evicted. When the champion state file is missing while the cache holds checkpoints (for example a wrong working directory), nothing is evicted either. Relative paths resolve against the process's working directory; run both services from the deployment dir, or configure absolute paths. Knobs:[validator] eval_ckpt_cache_dir(_eval_ckpts),eval_ckpt_cache_max_gb(120),eval_ckpt_cache_min_free_gb(60) andeval_ckpt_cache_grace_seconds(3600). - a checkpoint referenced by the validator state,
-
Harvest saves are atomic. A harvest is extracted into a
.partial-*sibling, fsynced, then renamed into place. A partial copy never looks complete. The pod's sha256 listing must match before the pod is released. On an operator pod, a copy that still differs after its re-harvest is retried, never uploaded. On a payer pod, a mismatch that persists after one retry is a terminal but unburnedcheckpoint_unverifiable. Flaky hardware, a write race or our own bug look the same as tampering. The box copy and both manifests (<harvest dir>.unverifiable.json) are kept as evidence and never evicted, with one critical alert. Any burn is the owner's manual call. -
A local-disk error is our fault. ENOSPC, EDQUOT, EROFS, EIO or EFBIG anywhere in a leg is classed
operatorand never burns a miner attempt. It halts rents at once. The partial copy is dropped, space is reclaimed, and the harvest is retriedharvest_disk_retries(2) times with doubling waits fromharvest_disk_retry_seconds(30). If R2 is configured, the last resort is pod → R2 directly. Otherwise the pod is kept (above).
Every pod rented on a miner's key is recorded per submission (ref +
funding; a re-fund starts a fresh count) in _train_work/miner_spend.json.
Each record holds:
- the landed $/h (else the miner's or the funded price cap);
- start and end;
- the outcome: a harvested, verified result, or waste. Waste covers a lemon, a lost or unreachable host, a crash, an our-side fault, or a pod kept during our re-harvest that never produced.
The queue's public view shows miner_spend_usd / miner_waste_usd, rounded
dollars. Before every rent on a miner's key (a fresh leg, a retry, a
lemon re-rent, a retrain after a loss), waste at or above [round] miner_waste_cap_usd (30; 0 = off) pauses the entry as paused_spend_cap:
- It is never failed, burned or rejected. The payer-key TTL, era rollovers and the attempt cap don't touch it. Its reveal keeps its seniority: the only reveal check is the fixed competition opening, never reveal age.
- One critical
miner_waste_capalert lists the spend, the waste and the pods. - Infra / our-side waste files a
spend_cap_make_goodapproval. Approve (approve-operator-leg <hotkey>) and the leg finishes operator-billed through the normal gates; it never runs on the miner's key again. Deny and it stays paused until a miner re-fund. - Miner-caused waste is a pause only; a re-fund resumes it. Generator faults still fail by the normal rules.
resume-operator-leg <hotkey>resumes and resets the count. An unreadable ledger fails closed: the entry is paused, nothing rented.
The miner's own total budget. cascade fund --max-total-spend USD (lane
A; header X-Max-Total-Spend, signed, positive) is stored on the entry as
max_total_spend_usd. A re-fund without the header keeps it; 0 clears
it. Before every miner-key rent,
all spend so far plus the next leg's projection must fit. Otherwise the
entry parks paused_budget, with the same never-reject guarantees as the
waste cap and one alert. A running leg is never killed for an overrun;
Lium key budgets are the hard backstop. A re-fund with a higher budget
resumes it, as does resume-operator-leg.
The payer-balance check (payer_projected_rate_per_hour) reads the miner's
balance with their vaulted key (never logged). A miner-chosen type is
projected at the miner's own cap, else the funded ceiling. An unknown
balance is retried and then the rent proceeds. A shortfall requeues
unburned as payer_balance ("top up: need ≈$X, balance $Y"), never counted
against any cap.
Empty account vs empty market (payer_account, 2026-11-04). A funded
leg's capacity wait probes the PAYER's listing; when that stays empty for
PAYER_CONTROL_AFTER_ZERO_POLLS (3) consecutive polls, the trainer runs
the same read-only listing on the OPERATOR key under the miner's price caps
and GPU-health guard, at most every PAYER_CONTROL_INTERVAL_SECONDS (300).
PAYER_CONTROL_HITS_NEEDED (2) consecutive controls that show capacity
while every payer poll in between saw nothing end the wait with a terminal,
unburned payer_account verdict ("your account lists no 8xH100 executor …
while the marketplace shows N …; balance $X / the key cannot read its own
balance") — the miner fixes the key/account and funds again; the entry is
never re-dispatched. A payer listing that fails with auth-class text
(401/403/"invalid api key") convicts at once. A control that is empty too
keeps the wait a market no_capacity. On a king's own key the verdict
counts as a key failure (the operator takes the leg over, DEC-CA-0055).
Every empty lium ls now logs one line with the raw row count and what each
filter removed (lium (payer key): 8xH100 listing — 0 raw row(s) …), and a
probe exception in the wait is logged instead of swallowed. The constants
live on the trainer mixin (runner_funded_pods.py), not in chain.toml.
The new knobs need no TOML change. Every default above is live as
shipped. Integral floats such as 3600.0 are accepted for every
seconds/count knob.
Deploy order (the 2026-11-02 money-safety release):
- Restart the validator first, or together with the trainer. Its eval cache retention and pins must be live before the trainer starts reclaiming. Expect a one-off trim of the cache to the budget and floor.
- Restart the trainer, the intake and stop opbill_guard.py
together. Older-code writers of the queue drop new fields; the pause and
approval parks live in the status itself and the fault count in the phase
history, so nothing un-pauses. Still, do not leave old writers running.
The per-leg approval gate replaces
opbill_guard.py; retire it. - Run
cascade-trainer list-operator-legsand approve each pending leg you want to pay for. Legs already in flight on the operator's bill are adopted with no approval.
The checked-in mainnet and testnet presets allow one active miner job per chain-resolved coldkey across A/B/C. The shared gate covers generation, capacity waits, training and fallback; king work has reserved admission. Unknown or stale ownership waits for fresh chain metadata. An already verified upload can finish without acquiring another compute slot.
Both presets also avoid a GPU type for three hours after two distinct pods of
that type fail within one hour. gpu_health.json preserves this across restarts.
Account, rate-limit and sold-out responses do not count as hardware failures.
A first transient host fault gets a 30-minute exclusion; repeated host faults
get three hours, while a rejected stale runtime can warrant 24 hours.
The independent operator watchdog records low or unknown account balance,
oldest queue waits by lane, held king settlements, repeated king failures,
GPU cooldowns and chain timestamp drift. A missing Shadeform balance reader is
reported as unknown. operator_health.json stores active/resolved events and
an outbox; unsuccessful delivery retries the same event ID. Set
ops_alert_webhook_env to the name of an environment variable containing the
HTTPS receiver URL. Without it, events remain available locally and in logs.
Receivers should deduplicate IDs. Public operations telemetry includes its
sample time and stale flag, never account keys or exact balances. A hung sampler
does not spawn additional samplers. An advancing but stale RPC head is
reconnected and refused until its timestamp is fresh; block heights still
determine eras and settlements.
These [round] controls require the matching installed release:
| Controls | Purpose |
|---|---|
ops_max_active_per_coldkey |
Shared owner limit; 1 in the presets. |
ops_gpu_failure_threshold, ops_gpu_failure_window_seconds, ops_gpu_cooldown_seconds |
Persistent GPU-type outage detection; 2 / 3600 / 10800 in the presets. |
ops_alert_interval_seconds, ops_alert_webhook_env |
Watchdog cadence and optional delivery; interval 0 disables it when the operator pool is off. |
ops_low_balance_usd, ops_oldest_wait_seconds, ops_settlement_hold_seconds, ops_clock_drift_seconds |
Alert thresholds; timestamp freshness also gates the trainer's chain head. |
ops_operator_providers |
Ordered operator accounts, using LIUM_API_KEY / SHADEFORM_API_KEY in the trainer environment. |
ops_operator_leg_cap_usd, ops_operator_daily_cap_usd |
Required positive limits before enabling the operator pool. |
ops_paid_fallback_after_seconds |
After a known payer-capacity shortage and this much total queue age, permit capped operator rental. Zero disables it. An explicit miner GPU choice is preserved. |
ops_king_backup_after_seconds, ops_king_backup_cap_usd |
Delayed operator-paid king replica and its additional cap; zero delay disables it. |
Provider fallback and king replicas ship disabled. The operator must supply per-leg and daily limits before enabling them. The spending journal reserves a conservative amount at the hourly price ceiling for training, startup, harvest and any declared hold. Closed attempts still consume their reservation; retries cannot reset the daily or leg cap. These are allocation limits, not a guarantee against a cloud provider continuing to bill after an unsuccessful deletion. The watchdog reaps expired allocations while the trainer is running; independent provider-side billing alerts remain necessary for a complete control-plane outage. The capped operator pool requires that watchdog to remain enabled.
operator_spending.json binds each create intent to its original account and
physical instance ID. Operator credential snapshots stay under
operator_credentials/, using the configured vault sealing key and local
permissions. Preserve both directories with the rental ledger. An uncertain
create retains its reservation and blocks a second create for that job, even
when a subsequent listing is empty. Resolve that outcome with the provider;
do not delete journals to reset spending. An unused failing account can be
skipped only when no unresolved or legacy allocation needs inspection there.
Shadeform deletion is checked against physical IDs, not instance-name prefixes.
Miner-funded rentals keep the same fail-closed rule in the payer vault's
create-intents/<lease>.json (non-secret: scope, provider, state). Such an
intent holds only that round's leg; it never expires and no command clears it.
After confirming with the provider, on the miner's account, that the create
did not allocate (or after terminating what it did allocate), delete that one
file. Never delete it while the provider might still list the allocation.
The king replica uses the same pinned generator, seeds, contract and init on a separate allocation. Where the primary's allocation is known, the backup must use another provider or GPU type. The first valid training completion wins; evaluation scores do not choose the replica. The choice is durable, and late callbacks only clean up their own allocation. Operator backup work does not establish a king-key lapse; a proven primary-key failure retains the existing funding/slash classification. King-only settlements still require a valid king result; redundancy reduces that dependency's failure rate.
Continuation ships disabled and is a release change. Declare
continuation_protocol = "cascade-continuation/1" in the applicable
[v2.training] or [marketcast] contract, qualify the worker image, then set
[round] ops_continuation_image_digest to that image's immutable digest and
ops_continuation_interval_seconds to a positive cadence. Detached dispatch is
required. Size ops_continuation_max_bytes for the full model, optimizer,
averages and any pinned MarketCast data; the default bound is 16 GiB. The
empty protocol preserves earlier training digests. An overlay does not install
this code or activate a release-only contract field on an older validator.
A snapshot follows a complete optimizer step and contains weights, optimizer
state, random generators, routing/averaging state, consumed data position and
remaining active training budget. It uses JSON and safetensors, with atomic
generation pointers and digest checks. The orchestrator periodically pulls the
committed generation to _continuations/ and restores its last verified copy
on a replacement pod. Failed transfers leave the previous copy intact.
Credentials never go to the worker. Copying a warm-start weights file alone
does not resume a leg.
The protocol also records a yielded shard series before returning it to the trainer. Earlier shard streams account for it only on the next iterator read; that historical behaviour remains available for replay. The declared protocol makes the final consumed series part of the new receipt digest without reading an extra series. Trainer, local scoring and audit replay use the same rule.
Recovery replays completed input batches without optimizer updates and checks their digest before restoring the saved random state. MarketCast also restores the original training snapshot instead of downloading revised bars. A different base, recipe, selection, seed, contract, dtype or qualified runtime is refused. Exact continuation requires the same GPU model; moving to an unqualified type must not silently reuse optimizer state. Retried work retains the operator-billed restriction. Before the first durable off-pod copy, a lost pod still requires a full restart; later losses can repeat work since the last copied boundary.
This protocol counts active training wall across attempts. Snapshot I/O, reconstruction and replacement startup are recovery overhead; they do not add optimizer steps or token budget. Qualify the cadence, transfer size, available local disk and provider hold before activation. Completed-job mirrors need operator retention management; the per-snapshot bound is not a total disk quota.
Run this offline smoke check in an empty temporary directory:
python -m cascade.trainer.qualify_continuation --tiny --device cpu --work-dir /tmp/cascade-continuation-checkIt compares fresh-process uninterrupted training with a worker that exits
immediately after a durable snapshot and is resumed in another process. It
checks cold/warm A, selected-expert B and annealing. The JSON report explicitly
labels the tiny CPU scope. For qualification on an already available GPU, omit
--tiny, select --device cuda and the deployed --dtype, and use a new work
directory. This command neither rents nor accesses storage services. It does
not qualify MarketCast's real Timestral base, cross-provider SSH lifecycle,
long-budget throughput, concurrent rentals, or recovery after a complete
orchestrator outage. Exercise those on testnet before enabling the policy.
CPU recovery resolves Targon launch intents using the documented, paginated exact-name VM listing. Empty/error/malformed listings retain the intent and prevent a second launch. Targon does not sweep unrelated workload names. See the provider API.
Funded B and MarketCast C recovery can use configured operator GPU lanes
through the same admission gate and lane pool as A. The fallback retains B's
selection/base or C's recipe/base and harvests through the existing dispatcher.
It never rents an additional operator pod. Enable the existing
[round] funded_operator_fallback and provision sufficient operator final lanes
before enabling recovery. With no lanes, recovery stays pending; a full pool waits
at most 30 minutes before requeuing. This is make-good compute, not a monetary
refund or a guarantee of recovery throughput.
Worker checkpoints include training_measurement.json, bound by the
immutable checkpoint pointer. It binds generator, corpus, effective contract,
warm start and measured training-budget points. The lineage counter excludes
the fork-anneal tail and caps batch overshoot at the contracted budget. It uses
the contract's denomination (series_points and scalar channel points are
different). Each completed seed epoch is durably registered before the rolling
scheduler retires it; counter and epoch receipt update together. Replay after
a scheduler crash cannot count the epoch twice. Missing/incompatible evidence
stays pending and is retried; unsigned dashboard telemetry cannot open the gate.
These are trainer-produced measurements, not independent hardware attestation.
Back up the complete trainer and validator state bundle before upgrading.
On first load, the old budget-proxy count moves to legacy_budget_tokens and
does not satisfy a closed gate's measured threshold. A gate already latched
open is not automatically closed. Old checkpoint measurements are not invented
from their budgets: reconcile historical epochs from retained evidence or wait
for newly measured epochs. Review this explicitly if upgrading a live network.
Discovery exposes the meter type, legacy estimate and unmeasured epoch count.
Build/pin a matching worker image and code fingerprint before deploying; old
workers produce no new measurement and will leave epochs unmeasured.
Submission tracking requires the matching CLI, intake and trainer release.
Run cascade discovery-check --json; submission_tracking must pass. The
trainer publishes a fresh status/v2.json heartbeat at its existing interval
even when the block/policy body is unchanged. Queue responses carry as_of and
use Cache-Control: no-store; C's trainer status now includes funded_queue
alongside the existing count summary. Agents can use fresh public queues or
request intake queues directly with cascade status --hotkey ... --lane ... --ref ... --intake ... --json.
Each admitted attempt has a unique attempt_id and bounded phase_history
(16 events), phase_at, requeue count and retry timestamp. Updates require the
same hotkey/ref, admission block, funding time and attempt ID, so late callbacks
cannot relabel a new attempt. Stages describe preparation, compute waits,
allocation/startup, training dispatch, harvest and settlement wait. They contain
fixed reason codes; provider exception text and credentials are not published.
Telemetry write failures cannot change the training outcome. Phase changes are
not worker heartbeats or token progress; a healthy intake does not prove a worker
is alive. Old in_flight entries remain explicitly unclassified until new work
reports a stage. Timelocked pending reveals remain count-only.
B and C also publish bounded submission_progress rows keyed to the funded
attempt. B persists qualification observations separately from fault outcomes;
only committed acceptance is reported as accepted. C retains exact trained
entry receipt observations, including their original boundary, across omitted
later receipts and restarts; older receipts cannot roll observations back.
Neither projection sets weights or changes qualification/merge/slot rules.
The CLI requires a fresh scientific feed independently of the intake, and
reports reward_status=not_verified. These are unsigned observations, not a
replacement for signed receipts or proof of on-chain payment. Old entries
without an attempt binding remain unknown rather than inheriting a result.
Publish the updated llms.txt and miner guides with the release. This metadata
does not change emission, admission or scoring rules and cannot resolve an
actual provider capacity shortage. Qualify the recorded stages, automatic retry
and stale-feed behavior through an isolated lifecycle before publishing live.
- Intake.
cascade-intake --queue-path <work_root>/funded_queue.json --vault-dir <work_root>/payer_vault --submission-dir <work_root>/submission_vault --chain-toml chain.tomlbehind a TLS terminator (see Fronting the intake).--queue-pathis REQUIRED and must be the exact path the trainer resolves for[round] funded_queue_path(work_root-relative) — there is deliberately no default, because a CWD-relative fallback split-brains the queue (funds return 202 but never enter). Likewise--vault-dir=[round] payer_vault_dir,--submission-dir=[round] submission_vault_dir, and--ttl-hours=[round] funded_entry_ttl_hours(36 h). Queue writes are flock-serialised via a siblingfunded_queue.json.lock— keep both on one local filesystem (flock over NFS is not a lock). Loopback bind is the default; the key header must never cross the wire in clear. The vault and submission dirs stay off git, off rsync'd pod trees, and off backups that leave the box. NO--trust-refsand NO--no-require-signatureon mainnet, ever. - Vault seal key. Set
CASCADE_VAULT_KEY_FILE(32 raw bytes or 64 hex, e.g.head -c 32 /dev/urandom > /root/.cascade/vault.key; chmod 600) to the SAME file for the intake and the trainer. - Trainer config (
[round], trainer restart; no consensus impact):funded_pods = "rent"(every challenger leg on its payer's own key),funded_king_rent = true(the trainer rents, ledgers and sweeps the operator king pod itself; requires[training] expected_gpu = ""),funded_sku_per_leg = truewithfunded_pod_skus(the allowed GPU types; each leg rents the cheapest fitting executor), the price guardsfunded_max_leg_cost_usd/funded_max_price_per_hour(optionallyoperator_max_price_per_hourfor operator-account rents only; unset = the sharedfunded_max_price_per_hour) withfunded_sku_wall_seconds(measured wall per SKU; drives the per-leg cost cap and the latest safe start — re-measure when the budget changes),funded_cpu_blocklist,funded_host_bench_floor,funded_field_cap(legs in flight),funded_pod_image+worker_code_fingerprint(re-derive on every image re-pin), andfunded_pod_checkpoint = "harvest". - Publish the intake URL to miners and watch:
cascade queue --intake <url>,funded/latest.json,cascade-audit latest(funded-roster check) and the trainer log's admission / SKU lines.
The intake runs on the orchestrator — the box holding the trainer wallet and the private eval pool — so exposing it raw is not an option. The division of labour:
The front proxy does volumetric defence. Terminate TLS in nginx/caddy (or put the DNS behind an edge like Cloudflare) and configure, at minimum:
- per-IP connection and request-rate limits (nginx:
limit_conn/limit_req; the fund/withdraw endpoints are a few requests per miner per day — single-digit req/s per IP is generous), - header/body read timeouts (
client_header_timeout,client_body_timeout), - a body size limit matching the intake's ZIP cap (
client_max_body_size=--max-zip-mb, or ~1 MB if the intake is funding-only), - proxy buffering ON, so a slow client dribbles at the proxy, not at a handler thread.
The intake keeps in-app backstops so a missing or misconfigured proxy degrades gracefully instead of taking the orchestrator down:
--request-timeout(default 30 s): per-socket read/write timeout — a slowloris client gets its connection closed, not a parked thread (the stdlib serves with NO timeout by default).--max-connections(default 64): concurrent-handler cap; over it, new connections get an immediate503+Retry-Afterand are closed without spawning a thread, so a connection flood costs one small write each.--chain-timeout(default 30 s): deadline on the reveal poll — a hung substrate websocket serves the stale reveal table instead of pinning every handler slot behind one dead chain connection.--request-deadline(default 120 s): whole-connection wall clock enforced by a reaper — a drip-feed client sending one byte per interval defeats a per-op timeout but not this.--max-uploads(default 4): concurrent submit-body buffers (each is up to the ZIP cap in RAM).--per-ip-rpm(default 60) /--per-ip-connections(default 8): a sliding-window request cap and a concurrent-connection cap per client IP (429 + Retry-After;/healthexempt). Behind the proxy every connection arrives from the proxy's IP, so pass--trusted-proxy <proxy-ip>and the client is read fromX-Forwarded-For's last hop — never honoured from any other peer.- Identity floors: submits (and funds) require a hotkey REGISTERED on the
subnet (metagraph oracle, fail-closed 503 when the chain is unreachable);
each hotkey gets
--max-hotkey-mb(default 256 MiB) of stored submissions; ZIPs are capped by bytes, decompressed bytes, AND member count; signed timestamps are NaN-proof, strictly increasing per (action, hotkey), and the v2 signature binds the sha256 of the key header, so a captured request cannot be replayed with a swapped key. - Structural caps already in the request path: declared-length
413before any body byte is read, the signature gate before body buffering, and the streaming decompression cap in the store.
Per-identity limits (registration + quotas) and per-IP backstops together mean a single attacker needs many registered hotkeys AND many IPs to hurt the intake; the front proxy still carries the volumetric load. A minimal nginx front:
limit_req_zone $binary_remote_addr zone=intake_req:10m rate=2r/s;
limit_conn_zone $binary_remote_addr zone=intake_conn:10m;
server {
listen 443 ssl; server_name intake.example;
# ssl_certificate / ssl_certificate_key …
client_max_body_size 128m; # = cascade-intake --max-zip-mb
client_header_timeout 15s; client_body_timeout 60s;
proxy_request_buffering on; # slow bodies dribble at nginx, not us
location / {
limit_req zone=intake_req burst=10 nodelay;
limit_conn intake_conn 8;
proxy_pass http://127.0.0.1:8790;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 180s;
}
}Then run cascade-intake … --trusted-proxy 127.0.0.1.
A funded pod runs on the MINER's provider account: the payer has console access, so everything in the worker's environment is theirs to read. Rules, enforced in code:
- The pod is isolated:
RemoteHost.isolateddrops everyforward_envand the dispatcher's global extras (WANDB_API_KEY); nothing from the orchestrator's environment travels. Pulls are anonymous (the repos are public by design). - Credential-free by default (
[round] funded_pod_checkpoint = "harvest"): the worker runs--local-only— it trains and leaves the checkpoint on disk, uploading nothing, and the pod's environment holds NO credential of any kind. The orchestrator then pulls the checkpoint over the same pinned ssh (harvest_remote_dir, tar streamed through Python'starfilefilter="data"so a hostile archive with../absolute/symlink members fails the harvest rather than writing outside it), runs the ingest guard on the local copy (deviation ⇒tamper), and uploads it under the OPERATOR's identity — the manifest pointer is a checkpoint whose bytes the orchestrator inspected before anything else could fetch them. A harvest transport failure settles as infra (requeue), never tamper. Requires afunded_pod_imagebuilt from a--local-only-aware release.- Pod released before the upload (
[round] funded_release_after_harvest = true, the default): once the box copy matches the pod's own listing of the checkpoint dir (every regular file, size + sha256; one re-harvest on a mismatch) and the ingest guards pass, a payer or operator-rented leg's pod is torn down and the Hippius upload runs from the box copy (pod released after verified harvest (…) — upload continues from the box copy). The copy stays on the box until the upload returns a pinned ref; a failed upload leaves the pending-upload record the requeued leg re-uploads from. A pod the trainer still benches on (payer-pod bench armed outside a DEC-CA-0063 validator-bench era), a shared operator lane or the era king pod is never released early; any listing failure keeps the pod to leg end. - Pin
funded_pod_imageasrepo:tag@sha256:…, not digest-only. Lium's launch API rejects the canonical digest ref and the client degrades a digest-only pin to the repo's floating tag (:latest), so a host with an OLD image cached under that tag boots stale code while the env digest gate still reads "pinned" (it attests the requested ref, not the runtime). Give the pin an immutable version tag so the degraded form stays on-release, and the trainer functionally probes each rented pod'sworker --helpfor the dispatch flags (and hashes its files againstworker_code_fingerprint) before any bytes move — a stale pod is requeued as infra, never burned.
- Pod released before the upload (
"robot"mode (fallback for a pre-harvest image only): the pod pushes with a Harbor robot scoped to the checkpoint project withrepository:pushonly — no delete, no other project, no S3, no HF:- Per-pod robot (preferred): minted at rent, revoked at teardown and by
every sweep, Harbor expiry
[round] funded_robot_duration_daysas the backstop. Harbor forbids robots managing robots, and the operator's everyday Hub identity IS a project robot (robot$cascade+cascade-bot), so minting needs a project-admin USER login on the orchestrator:CASCADE_HUB_ADMIN_USERNAME/CASCADE_HUB_ADMIN_PASSWORD(used only by the minter; never forwarded anywhere). - Static funded robot (fallback): a push-only robot you create in the
Hippius UI, handed to the trainer as
CASCADE_FUNDED_HUB_USERNAME/CASCADE_FUNDED_HUB_PASSWORD; rotate it by hand. A user login in that slot is refused. - Neither → the leg fails CLOSED (skipped, unburned). The operator's own Hub login is never an option.
- Per-pod robot (preferred): minted at rent, revoked at teardown and by
every sweep, Harbor expiry
- Pod identity is pinned for the leg. Names are owner-chosen and
reusable, so the trainer records the platform's pod id at rent and the
container's SSH host key (generated at first boot), dispatches with
StrictHostKeyChecking=yesagainst that key only, and re-checks the platform id before dispatch and when the leg returns. A pod relaunched under the same name, or a different container answering at its address, settles the entry astamper— miner fault, terminal, hotkey spent — never as infra. These checks bind the observed pod identity; they do not attest that a miner-controlled host executed training honestly. Keep the harvest/checkpoint checks and credential isolation even with pinned SSH. - A checkpoint is data, never code. A miner's pod produces the
checkpoint, so no scorer may execute anything shipped inside it.
cascade/eval/checkpoint_guard.pyrequires the checkpoint'sforecast_wrapper.py/model.pyto be byte-identical to the release's own copies (any other.pyis refused),config.jsonto equal the contract's model config, and the safetensors HEADER (names/dtypes/shapes) plus file size to match the pinned model — before a tensor is allocated. The trainer runs the guard on every funded leg's checkpoint before it can enter a manifest (a deviation settles astamper); validators, audit and bench run it again on their side. - The payer vault is sealed at rest when
CASCADE_VAULT_KEY_FILEis set: AES-256-GCM per entry with the hotkey as associated data, so a copied vault directory is useless without the key. Unset = plaintext 0600 with a startup warning — never run mainnet that way. Plaintext entries still hydrate and re-seal on their next insert.
Trainer-only; nothing here reaches a manifest or a validator verdict.
- Operator runway guard. committed = Σ rate × remaining expected wall of
every live operator pod on the funded ledger (king pods, backups,
operator legs; the rate booked at launch, else the ceiling) + each
operator_standing_pods_usd_per_hourentry ×operator_runway_alert_hours. runway = (balance − committed −operator_balance_reserve_usd) / burn. Underoperator_runway_alert_hours(6): one criticaloperator_runway_lowalert. balance − committed − the launch's projection < the reserve: no new operator rents (operator_rents_held, distinct from the 402 breaker) — every operator launch paid from the account the balance belongs to (Lium: the king's included) is refused right before the create; it clears by itself. A CPU VM billed to another provider (Shadeform) is not held by the Lium balance — its own 402 breaker and the daily cap still apply. A launch refused by any of these gates rented nothing and does not count toward the per-hotkey CPU rent cap. - Every operator allocation is booked in
operator_spend.jsonat its landed $/h × expected hours (lemons too, settled to their real cost at teardown), and the daily cap counts it peroperator_daily_cap_scope:"miner_legs"(default: operator-billed miner legs),"all"(+ CPU VMs, lemons, other operator pods — never the king / backups) or"all_incl_king"(everything). The king pod and its backups are never REFUSED by the cap; the runway guard covers total spend. A king launch is judged at the best live offer and held only when balance − committed < reserve; when only its own projection does not fit, oneoperator_runway_kingalert (it rents anyway). - Priced keep (above), the lost-run retrain is operator-billed, an unreachable pod last seen with a finished unharvested run is kept for the priced keep, not the 30 min unreachable grace.
- Post-rent rejects (short pod, stale image, slow host, NCCL) and lemons
on a miner's key are booked
make_goodin the miner spend ledger (spend, never their waste) with oneminer_make_good_owedalert carrying the amount. A short pod's host IP is quarantined and both it and the executor go on the persistent short-pod blacklist (below). Listings are filtered on the exact GPU count before renting. A listing without a whole-pod price is priced at per-GPU × GPU count, never per-GPU. - Waste alert. The first infra loss taking a submission's infra waste to
miner_waste_alert_usd(15) files thespend_cap_make_goodrequest at once. A leaked pod's spend record stays open until the sweep confirms it gone. - Backoff. An unburned
infrarequeue (and any other unburned requeue that is not a market wait) backs off; a lemon verified gone resolves its leg's create intents; a payer's key and leases are re-stamped while it has ledger rows or open intents. - Pre-corpus gates. The money gates run as a side-effect-free would-rent verdict before the corpus, so no CPU VM is paid for a leg that cannot rent; the operator rent path has the doomed-leg check.
- King pods.
king_pod_ttl_margin_seconds(0 = off) sets a Lium--ttlof one era + the margin at create (Lium cannot extend a TTL). The idle reaper (king_idle_reap_checks= 2, everyking_idle_check_seconds= 900) terminates king BACKUPS and stale-era king pods that are not the king host in use and hold no live worker / unharvested run on consecutive checks, after a final on-pod check, with one alert. The in-use king pod is never reaped (the era's benches run on it). - Inventory. Every
operator_inventory_interval_seconds(3600) the OPERATOR account'slium psis diffed against the funded ledger, this deployment's funded names,operator_inventory_allow, the standing-pod names and every host IP / name inoperator_inventory_host_files; oneoperator_unknown_podalert per unknown pod with its name and $/h. Nothing is ever deleted.
-
A miner-caused payer-pod loss is the miner's. A generic loss of a payer pod asks the PAYER's account first: only a balance ≤ 0 proves a payer-side cause (
payer_side: requeued UNBURNED with the reason shown, never failed, never operator-billed). A pod listed stopped / terminated / absent is NOT proof (platform terminations, host failures, our own TTL):payer_pod_lost, requeued unburned on the miner's key with no make-good and no quarantine. An unreadable GPU-process cmdline (host-namespace PIDs) is unknown, never foreign. A genericinfraloss never setsoperator_billedon its own (only an operator-class verdict with our-side evidence does). A throughput / first-step abort of a payer pod probes for foreign GPU processes first (a miner's GPU hog ⇒payer_side, no host quarantine). Every approval request recordslast_error,error_classand thatevidence. -
Bounded ssh harvest. A receipt's
checkpoint_dirmust be relative to the pod workdir (no..) or inside a_train_worktree; the stream stops at the model size's expected checkpoint (expected_checkpoint_gb_by_preset,expected_checkpoint_gbthe floor) ×harvest_max_bytes_factor(1.2). On a PAYER pod a breach is a terminal, UNBURNEDcheckpoint_unverifiable; on an OPERATOR pod it is an operator fault that keeps the pod (our model grew — raise the table). Never a disk fault. -
Persistent short-pod blacklist (
_train_work/short_pod_blacklist.json, flock'd). A short-pod reject (fewer GPUs visible than the layout rented) blocks the executor id AND the pod's host IP for 72 h; a second strike within 30 days, 7 days. It applies to every leg, payer and operator account, and survives restarts, becauseLiumProvider._list_executorsfilters it before any rent.cascade-trainer rent-statusshows it.cascade-trainer block-executor <id|ip>(7 days by default) andunblock-executor <id|ip>edit it. Lium's listing has no free-GPU field, so a short machine is only found by renting it; the blacklist keeps it out after that. One-off seed from the trainer.log findings (29 short-pod rejects since Oct 25):cascade-trainer block-executor 3bb98239-513b-4e07-a577-4e3ffa1004d7 --reason "golden-lion-db: 4/8 GPUs x4" cascade-trainer block-executor 63cadadb-b379-4c45-8a01-04a81b135933 --reason "1 GPU x3" cascade-trainer block-executor f092246d-5704-4c9e-b57f-7c0710a06deb --reason "3 GPUs x2"Never-ready lemons feed the same file. A funded pod rejected as "not ready within …" on a host-side fault (never an auth / balance / rate limit from
lium up, never a create the platform did not list) records a not-ready strike on its executor. Two within 7 days block the executor with the short-pod durations above (72 h, then 7 d on a repeat), with the reasonnever ready 2x in 7 d (not-ready lemon); the pod's host IP follows the same rule (blocked only once two distinct executors behind it are). One strike alone blocks nothing beyond the existing 3 h bad-executor mark and 0.5 h host quarantine.unblock-executoralso clears the count. -
Post-rent rejects / lemons never propose a full-leg operator make-good: their cost is booked
make_good(one priced alert, sized to the loss). A make-good owed is OUR debt to the miner and never counts toward the waste-cap pause (that gates on genuine waste: miner-side + infra host losses); the per-attempt bad-pod budget and the short-pod blacklist bound a bad-host streak. -
Spend is booked per RENT, never per pod name (names are reused across rents): a fresh create opens its own record (
rent_id), and a pod found lost ("not live" at a re-attach, or still open when its name is rented again) closes at its LAST-SEEN-ALIVE time as infra waste. -
Withdraw of a live (
in_flight,in_round, parked …) entry is a 409 and keeps the key; a withdrawn entry is terminal (never requeued) and never gets an operator request. -
Rejected settlements (
receipt_watch_validator_hotkey): our own validator's rejected receipt of a settlement we published raises one criticalsettlement_rejectedalert naming its legs (read-only). -
Approvals without
--max-costget a ceiling of projection × 1.25; a landed cost above it releases the pod and re-opens the request. -
Daily cap reads a rolling 24 h window; each preflight reserves its projection atomically (flock), released at the launch / on a no-rent; every operator pod is trued up to its real cost at teardown.
-
Adoption of a live operator pod retries the probe 3× with backoff and never terminates a pod whose worker is alive.
-
Evidence copies (
.unverifiable.json) stay protected for 72 h and only the newest 2; the manifests are never deleted. -
King key: a rotated king key must also hold the king leg's projected cost (
GET /users/me), else oneking_key_unfundedalert. -
Stuck teardowns: a ledger pod whose teardown keeps failing > 15 min raises one
teardown_stuckalert with the accrued $. -
Knobs:
ops_operator_pool_enabled(false) switches the operator pool on; the spend caps no longer do.ops_paid_fallback_after_secondsno longer allows operator rents for every caller. An owner approval on file always allows that leg's operator rent;funded_operator_fallbackonly decides whether a non-miner fault PROPOSES an operator make-good (files the request) — turning it off no longer strands approved make-goods. -
Restart safety: a leg already on the operator's bill (its operator pod on the ledger, or in flight operator-billed and re-attached) keeps its operator recovery whatever
funded_operator_fallbacksays. -
Per-coldkey limit (
ops_max_active_per_coldkey) bounds only OPERATOR-paid work: the CPU-VM tier of every leg and operator-billed legs. A miner-key GPU rent never takes a slot, king legs are exempt, and an unknown owner uses the last persisted identity (no TTL) or proceeds. -
The CPU-VM reaper also covers providers a ledger row names that the config dropped. A harvest past its soft deadline is judged on its last-10-min rate. A lemon's spend record uses its landed price when known.
funded_queue.json now carries schema_version (2). Unknown entry keys
ride FundedEntry.extra through every load/save, and a writer whose schema
is older than the file's refuses to save (critical log,
QueueSchemaTooNew) instead of dropping fields. Writers deployed before
this field existed (the intake and opbill_guard on 5e3fade) cannot be
fixed retroactively and still drop unknown keys, so deploy in this order:
- stop
opbill_guard(redundant since #417); - restart
cascade-intakeon this release; - restart the trainer on this release (the validator does not write the queue).
Never run a pre-release intake or trainer against the queue again: it
cannot see schema_version and would still drop fields. From this release
on, a writer meeting a NEWER schema refuses to save rather than lose them —
bump QUEUE_SCHEMA_VERSION whenever a release adds an entry field that an
older writer must not drop.
The ssh harvest above tars each finished leg's checkpoint (~14 GB with
optimizer.safetensors and weights_stable.safetensors) onto the trainer's
disk under work_root/_funded_harvest. On 2026-10-31 several harvests at once
filled the 154 GB disk, teardown hit ENOSPC and pods kept billing miners for
hours. The r2 transport keeps checkpoints off the box entirely. It ships
dark: the default is "ssh", which is the old behaviour unchanged.
How it works (cascade/trainer/r2_transfer.py):
- The box plans a multipart upload on the Cloudflare R2 backup endpoint for
every top-level checkpoint file the Hippius push would carry. It mints one
presigned
upload_partlink per 512 MiB part (2 h expiry), or a single presigned PUT for a one-part file. - The pod uploads over the pinned ssh it already uses:
tail -c +OFF FILE | head -c N | curl -X PUT -T - "$url". Four parts run at once and each part is retried. Links go over stdin, never the command line. No operator credential reaches the pod: a link lets the pod write one part of one object until it expires, nothing else. - The box checks
list_partsagainst the plan: exact part numbers, sizes and the ETags the PUTs returned. Any mismatch aborts the upload. Then it completes the upload and re-checks the object size. - The box streams every object back from R2 and hashes it with sha256,
writing nothing to disk. The hashes must match the worker receipt's
in-memory
tensor_digests. A mismatch, or a tensor file the worker never wrote, istamper, the same verdict the ssh harvest gives. The contract guard (verify_checkpoint) runs on a sparse skeleton in the system temp dir: small files in full, plus each safetensors header followed by a hole of the real length. That is a few KB on disk. - The pod is torn down now, before any Hippius push — the early release
of
funded_release_after_harvest(above), with the R2 objects as the verified copy: the box asks the pod for its own listing of the checkpoint dir (size + sha256 per file) and every pushed file on R2 must match it, with nothing missing on either side (files the transport leaves on the pod — off the push allow-list or in subdirectories — are not compared). Thenpod released after verified r2 harvest (…) — Hippius push continues from R2. An unreachable listing or any difference keeps the pod to leg end (no re-upload); the knob off keeps it too. When the completion bench keeps the pod, it stays for the bench as before. - The Hippius push from R2:
- Relay host (
[round] ckpt_relay_host = "<name>"): an operator-owned VM listed as a[[host]]entry in the hosts file. The box sshes in and runspython -m cascade.trainer.r2_transfer relay, sending presigned GET links on stdin. The relay downloads into<workdir>/_r2_relay, re-checks every sha256, runsverify_checkpointandverify_weights_finite, pushes withupload_dir_to_hub_or_hf, returns therepo@sha256:ref and deletes its copy. - No relay: the same steps run on the box in
work_root/_r2_push, but only when free disk stays at or abovemin_free_disk_gbafter the download. Otherwise the push is requeued. - Requeue or failure:
<hotkey>--<job>.pending.jsonrecords the R2 keys, so the requeued leg pushes from R2 without a pod or a retrain. The objects are deleted after a successful push.
- Relay host (
Power-EMA shadow files (#400): a pyramid leg also writes
weights_ema005.safetensors, weights_ema010.safetensors and/or
weights_raw.safetensors. These are comparison data for the operator and are
never scored or published. They are verified with the rest of the checkpoint
(receipt digests and pod listing).
- r2 transport: the shadow files go to R2 under the leg's prefix. The Hippius push skips them, and so does the local disk check. They are kept when the published objects are deleted, and the bucket's lifecycle expiry removes them.
- ssh transport: once the box copy passes every guard, the shadow files are
deleted from it. With
[round] funded_keep_shadow_ema = true(default false) they are moved instead towork_root/_shadow_ema/<base_seed>/<hotkey>--<job>/. That directory is outside the harvest dir, so neither the upload nor the cleanup touches it. You must prune it yourself.
The published checkpoint keeps the pre-shadow file set either way.
Validators see no change: the manifest still carries the Hippius
repo@sha256: ref. A non-finite checkpoint is found at the push, after the
pod is gone, and settles as the miner's generator fault, as before.
MarketCast fine-tunes still use the ssh harvest.
Enable (testnet first, Approve-tier):
[storage]
backup_s3_endpoint = "https://<account>.r2.cloudflarestorage.com" # required for r2
[round]
funded_harvest_transport = "r2"
ckpt_relay_host = "ckpt-relay" # "" = bounded local push, else requeue
min_free_disk_gb = 30 # GiB; also the rent-time disk guard; 0 = off
ckpt_r2_bucket = "" # "" = [storage] backup_bucket (else manifest bucket)
ckpt_r2_prefix = "ckpt/" # objects: <prefix><base_seed>/<hotkey>/<job_id>/<file>Before you flip it:
- The worker image needs
curl,tailandhead. These are indeploy/Dockerfile. Rebuild the image from this release, then re-pinfunded_pod_imageandworker_code_fingerprint. - Credentials:
BACKUP_S3_ACCESS_KEY/BACKUP_S3_SECRET_KEYon the box only. The R2 token needs object read/write/delete and multipart on the bucket. - Canary one leg per stage on testnet. Watch the log for
r2 harvest from <pod>: N file(s)and check the pod was gone beforehippius push.
Relay host setup. Use a small operator VM with disk at least the largest
checkpoint (about 20 GB for the 1.23B preset) plus headroom, and good
bandwidth to R2 and Hippius. Install the cascade checkout at the released
commit (uv sync --frozen), so its checkpoint_guard matches the trainer's.
Add it to the hosts file:
[[host]]
name = "ckpt-relay"
host = "203.0.113.7"
user = "root"
key_path = "~/.ssh/operator"
remote_python = "/root/cascade/.venv/bin/python"
workdir = "/root/cascade"
stage = "heat" # accepted, never dispatched to: the relay is not a GPU lane
forward_env = ["HIPPIUS_HUB_TOKEN"] # or keep Hub creds in the relay's own envKeep stage = "heat". With "any" or "final", the relay would join the
operator lane pool and could be seated with a leg. forward_env values travel
to the relay on stdin, and only when the entry is not isolated. That is
allowed because the relay is operator-owned. Never point ckpt_relay_host at
a miner or payer pod. If the provisioner rewrites the hosts file, check that
the relay entry survives.
Bucket lifecycle. Add an R2 lifecycle rule on the ckpt/ prefix that
expires objects after 3 days and aborts incomplete multipart uploads after
1 day. Successful pushes delete their objects. The rule cleans up what is
left: tamper evidence, aborted uploads, and pending pushes whose era has
passed (the box ages out pending records after 3 days too).
Disk guard and ENOSPC-safe teardown (independent of the transport, so
they are live at the default): a NEW funded or operator leg pod is not rented
while work_root has less than min_free_disk_gb GiB free. The leg is
requeued as no_capacity and never burned. A re-attached or adopted pod,
and king backups, always keep going. Teardown no longer needs a disk write
before terminate: a failed lease TTL re-stamp, lease removal or ledger write
is logged and the pod is still terminated. The startup sweep settles any
ledger row left behind.
Rollback: set funded_harvest_transport = "ssh" and restart the trainer
between legs. Pending R2 records still push from R2 on their retry. In-flight
legs finish on the transport they started with.
This is the trainer-sourced benchmark path. Validator-sourced v2 A benchmarks skip payer sweeps and their verification; see the next section.
With [telemetry] funded_bench = true, a payer pod benches its own
checkpoint the moment the leg finishes — harvest → ingest-verify → bench →
tear down — and is never held to a settlement boundary. Expect ~1 h of extra
pod time per submission; [round] funded_bench_margin_seconds (default 0)
reserves time for it before a leg's target boundary. The bench data is
sideloaded from the trainer box and no HF token goes to an isolated host
(a failed sideload degrades to an anonymous download).
Payer numbers are a filter, never a published fact. A pod the miner
controls can write any benchmark_report.json it likes, and the signed
bench report feeds the DEC-CA-0013 promotion pool (the shared init for
every future era). So the top funded_bench_verify_top (default 1)
payer-reported challengers are re-benched on the era king's pod from the
orchestrator's guard-verified copy of the checkpoint. Within
funded_bench_verify_tolerance (2 % relative, one-sided) the operator's
numbers replace the payer's; a payer number better than the re-bench by more
than that is a forged sweep — the entry is dropped from the report (its
submission is spent anyway; the promotion seat is what the drop denies) and
logged as TAMPER. Unverified payer numbers (below the top-N) are logged
only. Everything the trainer signs was produced on operator hardware; the
payer bench decides only which challenger is worth the king pod's time.
BenchReport carries no provenance field (validators sign-check the
canonical body; adding one would fork them).
From [v2] bench_source_from_block (mainnet: "v2", i.e. the v2 go-live)
with [v2] bench_source = "validator", lane A is no longer benched at leg
completion. The trainer runs no challenger or king sweep, no payer-pod
verification and no bench hold for those eras, and no init bench either:
the main validator benches the init itself. For a random-init seed epoch the
trainer only publishes the init POINTER (a pointer-only role = "init"
report entry whose six scores are the -1.0 sentinel — published alone,
with no rows, and never scored: BenchEntry.pointer_only marks it,
entry_for_pointer skips it, the king check and the audit defer on it, and
older validator builds defer too, on the missing king entry and init rows): the random-init
checkpoint is built on the era's already-mapped operator king pod after the
epoch's king leg (CPU, moe_trainer.build_random_init run pod-side, refused
on a contract-digest mismatch), copied back as plain files, verified against
the pod's listing and uploaded. Nothing is ever rented for it, the box never
builds or loads it, and only an OPERATOR-billed pod is used (ledger row with
no payer; never a miner-key pod, never one with a leg in flight): the era's
operator king pod, else an idle operator lane (never the last idle one), else
the request is parked and the next operator-billed challenger pod of the seed
epoch builds it right after its leg's verified harvest, before its release.
While parked the king check defers with "no init pointer"; a failed build
counts against INIT_BENCH_MAX_TRIES per seed epoch. Past the cap the trainer
no longer holds a pod for it but keeps retrying every
INIT_BENCH_RETRY_SECONDS (1 h) until the pointer lands (the log says "GAVE UP
fast retries", then "slow retry N failed"; 2026-10-31 the old hard give-up
after three ssh timeouts left seed epoch 2611 without a pointer, so no king
check could pass for the whole epoch). cascade-trainer rebuild-init-pointer
(an optional --reason TEXT is logged) resets the counter of every era still
wanting its init and the trainer retries on its next tick (the request file
_train_work/init_pointer_rebuild.json is consumed). A warm-started epoch
needs nothing (the pointer is warm_start_ckpt).
The random-init repo is named ckpt-<base_seed>-<arch>-random-init-sb<seed_block>.
From [v2] init_epoch_bind_from_block (default 0 = off; release first, then
activate) the king check accepts a random-init pointer only when that suffix
names the settlement's verified era-stamp seed_block, and otherwise defers
with "no init pointer for this seed epoch". That closes the case where a new
epoch's first settlement reads the previous epoch's init from the previous
settlement's report.
The main validator (the wallet whose hotkey is [netconfig] owner_hotkey) does the sweeps instead, and only when they are needed:
- the challenger that cleared the private pool, plus the era king (its rows are cached per pointer, so it is swept once per king checkpoint);
- the king and the current init when the DEC-CA-0055 king check is due.
It signs bench_v/round-<id>.json (rows under bench_v/rows/<id>/) with its
own hotkey and publishes it before the settlement is marked handled, even
with no swept entries. When a clearer or due king check needs this evidence,
every evaluator, including the main validator, holds until a later
trainer-signed manifest acknowledges the exact report digest (or records
"absent" after the configured timeout). A quiet settlement that needs no
report can proceed without one. Satellites replay the bound evidence in the
main validator's receipt.
What the main validator needs:
- A GPU:
--device cuda, or an offload pod via--eval-hosts(the same path the gift gate uses,run_bench_via_host). - The benchmark sidecar (
[eval] benchmark_project_dir) and the benchmark data staged locally:uv run --project benchmarks cascade-benchmark-download --data-dir <dir>, then[eval] gift_gate_data_dir = "<dir>". Coverage and the time limit follow[eval] cascade_bench_max_series(0 = full battery) andbench_hold_max_hours. FEV is added when the DEC-CA-0061 rule is in force; it pulls its own data from the HF hub. That rule ("avg_noninf_winrate", rule v2 2026-10-20) reads the per-task rows only: average ratio < 1, no suite's ratio LCB above 1 +[scoring] bench_gate_tolerance(0.01), win rate >=[scoring] bench_gate_min_win_rate(0.5). Before arming it, runscripts/bench_seed_noise.pyon seed replicas of the recipe to calibrate the tolerance (DEC-CA-0061 "Rule v2"). - Write access to the manifest bucket. It already has this, because it publishes receipts there.
Budget: historical trainer timings do not qualify the current four-suite run
on Targon; measure the actual checkpoint, GPU and data first. The ceiling is
bench_hold_max_hours, which is 9 h on mainnet. The sweep runs inside the settlement, so
a winning settlement takes one sweep (king cached) or two (new king pointer)
longer to judge, and the next settlement waits behind it. Use
cascade_bench_max_series or a faster offload GPU if that comes close to the
3 h cadence. Each checkpoint's sweep has its own allowance, [v2] bench_sweep_hours (mainnet 2.0, its retry included); the king's sweep never
spends a clearer's. [v2] bench_budget_hours (mainnet 6.0) caps the
settlement: a clearer starts only while the cap still holds its whole
allowance (plus the king's when the king is not cached), otherwise it is
published budget_deferred — skipped on every validator (never a loss) and
carried by the trainer into the settlement that records the report (at most
2 carries, inside the era), where it is judged again.
A failed sweep is classified. When both attempts show checkpoint evidence
(non-finite forecasts, or every config failing at scoring) and nothing
environmental, the sweep is published checkpoint_fault, and a challenger
listed there fails the gate on every validator. Everything else is
published infra_failed: OOM / CUDA, timeouts, store / network, dataset
fetch, a sidecar crash, or an allowance spent before the retry. An
infra_failed clearer is skipped and never takes a loss. With nobody left
under enforce, the round is inconclusive and the king holds. Size
bench_sweep_hours from a measured sweep: an allowance that is too short
turns every sweep into an infra timeout. If the publish itself fails, the
settlement is rolled back and held until the next poll.
The main validator can rent its sweep pod on demand instead of keeping one
standing: cascade-validator --bench-lease bench_lease.toml --eval-hosts <the hosts_path it names> (template: deploy/bench_lease.example.toml).
The lease rents only when a sweep is needed (a clearer reaches the bench
gate, or the king check is due), bootstraps it with
deploy/bench_pod_bootstrap.sh (sidecar from benchmarks/uv.lock, pinned
data), publishes it to --eval-hosts and terminates it after
idle_release_minutes with no sweep (never while a benchmark process runs).
Miner safety:
- while the pod is renting, the settlement holds: no verdict, no report, and the private pool is not re-scored on every poll;
- the hold is bounded by
max_wait_hours, which the loader checks fits inside[v2] bench_report_timeout_settlements. Past it, a clearer is publishedbudget_deferred(never swept, never a loss, carried by the trainer) and a king / initinfra_failed; - a pod lost under a sweep is re-rented and the settlement holds again; a second loss before a good sweep goes straight to the deferral;
- rows carry a
hwrevision (GPU model, driver major, sidecar torch). A cached king swept on other hardware is re-swept, and the gate never pairs two fingerprints; - with a lease there is no local fallback.
Before arming: qualify every gpu_types entry with one measured sweep, set
[eval] gift_gate_data_dir to the bootstrap's data dir (/data/bench), and
keep the standing pod until the first leased sweep has published.
The sidecar runs each suite as independent work units (one GIFT-Eval config, BOOM dataset, TIME task or FEV task) over N worker processes, each loading the checkpoint(s) once. Several workers can share a GPU while others prepare data or compute metrics. Each retains every paired model; larger checkpoints therefore fit fewer workers. Qualify identical reports on the actual hardware before choosing concurrency. Only the main validator sweeps; every other validator replays its signed rows.
-
[eval] bench_workers = "auto"(or an int;1= sequential). Withauto, a short probe process loads the checkpoint(s), runs one warm-up batch at the sweep's batch size and reads the allocator peak. Per-worker VRAM is that peak × 1.25 + 0.5 GiB of CUDA context. The worker count is min(Σ per GPU ⌊(free − 2 GiB) ÷ per-worker⌋, effective CPU cores − 1, available host RAM / worker allowance, 16, units), spread round-robin over the visible GPUs with one GPU per worker (CUDA_VISIBLE_DEVICES).--device cuda:Kpins the sweep to GPU K. A failed probe falls back to one worker. The chosen plan is logged as a[sweep] plan:line on the sidecar's stderr. On an offload pod (--eval-hosts), the sizing runs on the pod. CPU affinity and container quotas apply; BLAS and torch use one CPU thread per worker. RAM sizing is conservative but estimated; the sidecar's--worker-ram-gbcan override it. -
[eval] bench_paired = truesweeps challenger + an unswept king (and king + init at the king check) in ONE pass, so each dataset is built once. The pass may run for both members'bench_sweep_hours. Its wall time counts once towardbench_budget_hoursand half toward each member's allowance, and it is each member's first attempt: a member whose report failed retries alone with what is left. A pass starts only while the cap holds both allowances. -
Measure before setting the DEC-CA-0063 budgets. On the sweep box (or pod):
python scripts/bench_timing.py <checkpoint> --data-dir <dir> --device cuda \ --suites gift-eval,boom,time,fev --workers auto,4 \ [--extra-checkpoint <king checkpoint>] --keep /tmp/cascade-bench-qualification
It runs the sidecar with
--workers 1and then with each setting you list, prints the wall time per suite and the speedup, and exits 1 unless every report is byte-identical to the sequential one. It also rejects unknown/partial data pins, missing suites and failed tasks. FEV's explicitly reported multivariate scope exclusions are allowed; they are not full FEV leaderboard coverage. Stage its local manifest before this run. All four suites are selected by default. Then set[v2] bench_sweep_hoursto cover the measured (paired) sweep plus its retry, andbench_budget_hoursto cover the clearers a settlement should judge. The current 2 h / 6 h values were set before any timing existed.
A prepared Targon GPU VM works through the existing SSH interface. This path does not automatically rent Targon GPUs; the automatic Targon adapter remains for CPU generation. Use the VM's assigned SSH address, port and login user, install this release and both benchmark extras with the checked-in lock, and stage all four suites on local disk. Do not forward wallets or provider keys.
Point cascade-validator --eval-hosts at its hosts.toml. Leave cuda_device
unset to expose all visible GPUs, or list one entry per GPU on that same VM;
the resolver combines those masks. It selects one physical machine, not a
distributed fleet. --device cuda runs on a GPU VM hosting the validator itself.
MarketCast currently uses the validator's own --device.
For an evaluation-only Targon VM, start the validator with both
--eval-hosts targon-eval.toml and --private-eval-hosts targon-eval.toml.
The private path covers A verdicts, king checks and B acceptance through the
shared evaluator; the public path covers GIFT-Eval, BOOM, TIME and FEV.
MarketCast C uses the validator's own --device and is not routed by either
hosts flag. Run the main validator itself on its intended GPU for C, or qualify
a separate C offload implementation before launch. A configured private host
that disappears holds evaluation. Public benchmarks retain the documented
local-device fallback when their host is unavailable. The repository does not
prove these flags are present in a running service.
Private-pool offload is separately opt-in through --private-eval-hosts;
see validator compute. Only histories
cross that boundary; targets and scoring stay local. Parallel private workers
use independent model copies and preserve the original per-window batch/RNG.
They default to one worker and require hardware qualification before enabling.
The worker's --qualify mode compares every sample with a sequential run:
python -m cascade.validator.private_worker CHECKPOINT REQUEST.npz OUTPUT.npz \
--device cuda --workers 2 --qualifyREQUEST.npz is made locally with cascade.validator.private_worker.write_request
using the effective training contract and selected windows. It contains no
targets or pool metadata. Keep this private fixture and checkpoint local to the
operator-owned VM. Save the returned timing/runtime JSON with the public
qualification results. A failed comparison must block enabling that mode.
Shipped OFF ([marketcast] from_block = 0). To arm it, type an era
boundary that is:
- ≥
[v2] from_blockand ≥[lane_c] from_block; - at or after
[v2] bench_source = "validator"takes effect; - set while
[netconfig] owner_hotkeynames the main validator.
The loader refuses anything else. Lane C is MarketCast-only (DEC-CA-0056,
owner 2026-10-20): the shipped [lane_c] targets are already the
mkt:<suite> targets of [marketcast] suites (mkt:returns, mkt:vol),
and the loader refuses any non-mkt: target ("lane-C competitions need a
forward-only eval — only MarketCast (mkt:) has one") and [lane_c] pool_prefix (retired; there is no pool_c and no cascade-pool publish --lane c). With lane C armed and MarketCast off, no target is active, so
lane C's share burns. Add a suite to [lane_c] targets only when it is in
[marketcast] suites. The loader also refuses arming while a class that lists
assets has an empty or unbuilt [marketcast.data_sources] entry. Empty the
class, or pick a built source.
Feeds (public, no key; used by the snapshot/training pipeline and optional audit reproduction, not by every validator to derive the recorded verdict):
-
Hyperliquid
POST https://api.hyperliquid.xyz/infois primary wherever it lists the asset:candleSnapshot1m andfundingHistory;- the main dex for crypto;
- the
xyzdex (xyz:<SYMBOL>) for FX, metals, commodities, indices, ETFs and mega-caps.
These are perp prices: they carry a basis and drift off-hours, which is acceptable for a forecasting eval.
-
Binance spot 1m klines are the crypto cross-check. Its result is reported in
snapshot.jsoncross_checkand never gates. -
dTAO alpha comes from
cascade-pool mkt-record-alpha --out <file>, cron'd every minute on a box with chain access. It reads the pinned SDK path (prices.alpha_price+SubnetInfoRuntimeApi.get_dynamic_info).publish --lane mkt --alpha-rows <file>folds the samples in. The reserve keystao_in/alpha_inare not verified against SDK 11.3.0; a missing key reads as missing. -
Open owner choices for FX, the other 25 crosses:
- Pyth Hermes polling by the main validator (redistribution terms unverified);
- Chainlink on-chain FX as a cross-check.
Neither has an adapter yet.
Snapshot (signed by the main validator's hotkey; validators refuse any other signer):
*/1 * * * * cascade-pool mkt-record-alpha --out /srv/cascade/alpha.jsonl
*/5 * * * * cascade-pool mkt-record --store /srv/cascade/bars
30 */6 * * * cascade-pool publish --lane mkt --alpha-rows /srv/cascade/alpha.jsonl \
--bar-store /srv/cascade/bars --wallet-name owner --wallet-hotkey main
50 5 * * * cascade-pool reveal --lane mkt --hf-repo <org>/cascade-eval-poolThe forward bar recorder is required (review M15). mkt-record
appends every CLOSED Hyperliquid 1m bar + funding sample to the bar store as
it closes; each pass re-fetches the last 48 h and merges, so an outage
shorter than ~3 days loses nothing, and an unrecorded minute stays missing
(never interpolated). publish --lane mkt builds the snapshot's minute bank
from the store and refuses to run without --bar-store (--allow-cold
only for tests). Start the recorder at least snapshot_days (21) before
arming: the config cannot see it, so arming cannot check it.
It carries the last snapshot_days (21) of bars on one UTC-minute grid,
plus a block → time anchor, plus a coarse HOURLY bank of
coarse_history_days (180) that every series forecast at ≥ 1h is cut from
(realized vol at ≥ 1d only: the std of the day's 24 hourly returns). It goes
to pool_mkt/snapshots/block-<N>.tar plus pool_mkt/index.json in the pool
bucket. Publish often: an entry's windows exist only once a snapshot
covering them is active. The verdict waits up to verdict_wait_blocks for
them.
Hyperliquid history depth (measured 2026-10-19 with read-only info
calls, BTC and xyz:GOLD): candleSnapshot serves only the most recent
~5000 candles of an interval — 1m ≈ 3.6 days, 5m ≈ 17 days, 15m ≈ 52 days,
1h ≈ 208 days, 4h ≈ 833 days (an xyz asset only since its listing), 1d
from listing. So:
- the hourly bank fits (180 < 208 days), and is what makes the 1d horizons scorable (21 daily bars could never hold the 32-step minimum context);
- the API alone fills only the newest ~3.6 days of the 21-day minute grid (less than the 6-day verdict wait) — hence the recorder above;
- a recipe's embargoed training range (≥ 30 days old) has no minute bars on Hyperliquid; the training snapshot also fetches the hourly bank, so ≥ 1h suites train on real data and sub-hourly ones only on the generator's.
Window eligibility and the verdict (review II, 2026-10-19): per (asset,
horizon) windows_per_cell (48) windows spread evenly over the eligible
span; a window needs ≥ min_bar_coverage (0.8) live bars and ≥
min_dollar_volume_24h ($1M; 0 = off) traded in the 24 h before it, and a
price window (returns, alpha) a context that is not flat. Nothing reads
the realised target: min_target_abs_bp was removed (red-team F3,
2026-10-20) and the loader refuses it. The bootstrap clusters are asset
class × cluster_block_hours (6 h). The gain is the geomean over (class ×
horizon) of the ratio of MEAN CRPS (red-team F1); horizons and asset classes
weigh equally; a verdict needs ≥ min_time_blocks (8) blocks. A class × block
× horizon cell whose base mean loss is ≤ zero_scale_frac × its horizon's
median is masked (counted on the receipt row as mkt.masked_cells), and each
ratio is clipped to cell_ratio_cap. The calibration gate runs on
mkt:returns AND mkt:vol. Per horizon it is two-sided on
coverage_gate_levels (coverage_tolerance below, coverage_tolerance_over
above). The tail hit rates at tail_levels must be within tail_tolerance
of 5 %, per horizon and pooled. Every rate is class-weighted like the gain,
and every band is widened to calibration_z standard errors. Tune the
liquidity floor once the recorder shows real volumes: it was not measured
per asset.
The loader refuses a configuration with an unscorable (suite, horizon): its
frequency's history (coarse_history_days for a ≥ 1h series, else
snapshot_days) must hold the 32-step context + the horizon, and the
forward wait (verdict_wait_blocks × 12 s − start_margin_seconds) must
hold two aligned windows (≥ 3 × the horizon). 5d was dropped from
mkt:returns for that reason.
Main validator:
- install the release's locked dependencies:
uv sync --frozen --extra train --extra hippius --extra chain --extra marketcast. The Timestral library is a git-pinned dependency in the committed lock. Do not regenerate that lock as a routine deployment step. - use a GPU (
--device cuda).
Every settlement registers its MarketCast entries (and every entry left pending) as ONE MarketCast round. For each round without a published report the main validator:
- loads the base at its pinned commit, sha256-checked before load;
- fetches each fine-tune, which must match the base's architecture, be
finite and carry a valid
marketcast_meta.json(covariates in the allow-list), else the entry gets a deterministicfault; - forecasts the base and the fine-tunes on EVERY eligible window of the
snapshot active at the round's block (base quantiles cached per window);
a compute failure (OOM, a library exception, the base not loading) is
retried twice, then the entry is marked
infra_deferred— never judged, never a loss; - measures each computed fine-tune against its SENIOR references (red-team
F4): earlier-committed entries of the same suite in the round and the
suite's slot holder, forecast on the candidate's own windows. It records
the closest reference as
near_dup, with that reference's forecasts as a second sidecar (marketcast/round-<id>/r-*.npy). This is one extra pass per reference not already forecast; - publishes the float32 quantile sidecars and then
marketcast/round-<id>.json, signed, public-read. It never overwrites a report; a missing snapshot or a failed publish is retried at the next settlement.
A non-infra exception raised inside a fine-tune's forward identically on
both attempts is that entry's fault (red-team F7). The infra shapes stay
infra_deferred: OOM, OSError, and messages naming CUDA / cuDNN / NCCL /
the driver.
Missing reports that have not yet been recorded can defer MarketCast work.
A round is applied
only when a later TRAINER-signed lane-C manifest records its report in
mkt_reports (the sha256 of the text the trainer read, or "absent" after
report_timeout_settlements (8) settlements); validators apply rounds
strictly in order and use exactly the recorded bytes. A recorded report
whose bytes cannot be read holds the whole settlement, including A/B, like
any bound read. Restore the exact recorded report and its sidecars from a
verified replica; never replace its digest, overwrite a report or make a local
decision to treat it as absent. Permanently losing all copies has no automatic
consensus-safe recovery in this release. Qualify report replication and restart
recovery before enabling C; C remains disabled in the checked-in mainnet config.
"absent" defers every
entry as infra; an entry deferred more than max_infra_deferrals (8)
times drops with the operator error class and the trainer requeues it
uncharged. Infra time never consumes an entry's verdict_wait_blocks. A
round no manifest ever records expires to absent after 2 ×
report_timeout_settlements MarketCast settlements.
Cost per round: one base pass over the windows, plus one pass per
fine-tune. Each pass covers windows_per_cell (48) × assets × horizons
windows. The cost on the validator GPU has not been measured (owner open
item).
Trainer: a MarketCast leg rides the lane-C queue and payer-pod path. The
pod runs cascade-train-worker --role mkt_finetune --local-only and needs
the marketcast extra in the worker image. The leg rents and harvests under
the throne contract reduced to [lane_c] leg_hours (mkt_leg_contract,
red-team F5), not the 5 h throne wall. The trainer stamps the leg's dispatch
time (train_time, unix) into the signed marketcast record — the
forward-only clock. It also stamps the dispatching commitment's coldkey
(red-team F6: validators count the Bonferroni family by owner), and keeps at
most max_open_legs_per_coldkey (1) MarketCast legs open per coldkey. The harvest checks the weights' sha256, the base's
architecture, the base's config.json, finite weights and
marketcast_meta.json (required fields, the pinned base, covariates in the
allow-list, ≤ 64 KiB). While MarketCast is active the trainer reads the main
validator's reports from the manifest bucket at every settlement and records
them in the lane-C manifest (mkt_reports), publishing a lane-C manifest
even when no leg finished.
Fine-tune checkpoints are uploaded under the trainer's Hub namespace like
every other checkpoint, which is public by design (cascade/shared/ hippius.py notes the namespace serves anonymous pulls), so a judged
MarketCast fine-tune is public-read from its harvest on — what
cascade-audit marketcast --reproduce needs.
With --submission-dir on the intake and [round] submission_vault_dir +
champion_publish on the trainer, code never touches a miner-hosted repo:
POST /v1/submit(body = the ZIP, signature over its sha256, optionallyX-Lium-Api-Keyin the SAME request) stores it in the operator-private store and returns avault/direct@sha256:<hex>ref + the exact chain payload — a byte-ordinary hippius commit, so no validator changes. A submit-with-key parkspending_revealand auto-queues when the chain reveal resolves (the intake sweeps on every request).- Ownership: earliest upload owns a digest; another hotkey committing your
digest is dropped at field entry; byte-copies still die at dedup, and so
does an entry that carries most of your unpublished module inside another
tree (
private_copy— identifier renames do not help; only published code is anyone else's to build on). - Publication: ONLY thrones publish (
champions/<digest>.zip+ index, public-read on the manifest bucket) perchampion_publish:crown(published when crowned; the shipped value) /delay(afterchampion_publish_delay_rounds) /dethrone. Losers never publish. - Pods: a dispatch stages exactly ONE entry's ZIP
(
SubmissionStore.stage_for_dispatch→ the pod's$CASCADE_VAULT_DIR); the king's published code also resolves via$CASCADE_CHAMPION_BASE({s3_endpoint}/{manifest_bucket}). Never rsync the whole store anywhere. - Key TTL: a submit-with-key entry parks
pending_revealand its key lives in the vault under the same TTL, so a reveal timed beyond the TTL fails the entry (the key it needs is gone). Keep[round] funded_entry_ttl_hours=cascade-intake --ttl-hours(36 h).
Costs follow the active contract and benchmark source:
| Work | Normal payer |
|---|---|
| A/B/C challenger training | Miner, through the selected rolling provider key |
| V2 A CPU shard preparation | Operator; the generation window is separate from GPU time |
| V2 king-ladder defense training | King, when a new leg is needed; a cached king can cover several eras |
| Required operator-caused retraining and failed-key fallback | Operator resources, with the applicable recovery/key-lapse rules |
| Validator-sourced A benchmarks and MarketCast judging | Main validator/operator |
| Earlier trainer-source payer benchmarks | Miner pod, with extra rental time; operator verification uses operator hardware |
Budget for the actual GPU wall, startup, harvest and applicable benchmark work; the configured reference-compute hours are not a marketplace invoice or ETA. Compatible completed work still needs signed publication and judging. Late, stale or deferred work follows the documented recovery rules.
- The wallet never leaves the orchestrator. Pods cannot sign; a bad pod can only return a checkpoint the validator's gates reject.
- Pods carry no credentials. Isolated,
--local-only, checkpoint harvested by the orchestrator. No secrets in the image;SSH_PUBKEYat launch, key-only SSH, no baked host keys. - Miner code is caged. Container sandbox or strict-netns subprocess, a no-egress firewall behind it.
- The runtime is pinned.
train_image_digest+CASCADE_TRAIN_IMAGE_DIGEST, checked before dispatch.
Storage-layer integrity (who can write what, and why a leaked bucket key buys vandalism but not a forged verdict) is in ARCHITECTURE.md.
Use uv 0.13.4 or newer, certifi 2026.10.19 or newer, and a patched OS CA store on the orchestrator, validators and every CPU/GPU worker. If using pip, use pip 26.2.2 or newer as well: it carries its own CA bundle. Upgrade through a trusted, patched installer; do not use an old pip or curl trust store to bootstrap its own replacement. Do not disable TLS verification. See the uv advisory and certifi release.
After refreshing OS CA packages, check each application's interpreter:
.venv/bin/python scripts/check_tls_trust.py
benchmarks/.venv/bin/python scripts/check_tls_trust.pyThe checker exits nonzero for affected trust anchors, old/missing certifi,
unreadable CA overrides or an uninspectable default store. It checks default
OpenSSL files and directories, certifi, Requests, botocore, pip's vendored
bundle when installed, and CA environment overrides. It makes no network
requests. Before installing dependencies, --system-only checks OS/custom
trust without requiring certifi. A clean result covers these stores in that
interpreter; it does not establish whether earlier traffic was intercepted.
Hosted CI uses scripts/prepare_ci_tls.py before dependency downloads. It
copies the runner's CA bundle into a job-local store, removes only the four
known affected root fingerprints, and points Python, Git and HTTP clients at
that store. The unchanged trust checker must then pass. This does not modify
the runner's OS store or replace patching production hosts.
The pod bootstrap transfers the orchestrator's patched uv over SSH and
checks its SHA-256 before use. The local binary and worker must use compatible
CPU architectures; CASCADE_UV_BIN can select the vetted binary. It checks
system trust before fetching Git dependencies, then checks each installed
environment. Eval dependencies install even without a local data cache;
failed installation or unsafe trust stops provisioning. Bootstrap refuses
uv system/custom/insecure trust overrides; ordinary application CA overrides
must pass the checker.
For a small benchmark-image trust check, build the tls-trust target before
fetching the scientific stack or datasets:
docker build --pull --no-cache --target tls-trust -f benchmarks/Dockerfile -t cascade-tls-check .Rebuild worker and benchmark images with --pull --no-cache; build both from
the repository root (-f deploy/Dockerfile or -f benchmarks/Dockerfile).
Qualify the resulting images on a disposable worker, publish immutable image
digests, and rotate the fleet using the normal contract/era rollout. Existing
image digests and running workers do not change when this source is updated.
Preserve queues and settlement state during replacement. Check the generation
CPU image too; an installed Python package update alone cannot patch OS trust.
Rotate whenever a pod ever held them. Trainer (.env, restart after): the
Hippius S3 pair (HIPPIUS_S3_ACCESS_KEY / HIPPIUS_S3_SECRET_KEY — read/write
on the manifest and logs buckets, read on the eval-pool bucket), the Hub login
(HIPPIUS_HUB_USERNAME / HIPPIUS_HUB_PASSWORD or HIPPIUS_HUB_TOKEN — push
and pull on the checkpoint repos), HF_TOKEN (fallback dataset write) and
WANDB_API_KEY. Validators need only READ access on the manifest bucket
(manifests/, benchmarks/, promotions/) plus write on their own receipt
prefix: issue scoped, expiring sub-tokens per validator and revoke any shared
pair. Each validator operator swaps the environment value and restarts.
The stakeholder scoreboard (cascade/website/stakeholders.html, served at
/scoreboard and /stakeholders) is deliberately short (owner 2026-10-10:
key models, tokens, competitions; little text). Three blocks:
- Hero — GIFT-Eval gap to the best. The king's latest GIFT-Eval MASE (toggle: CRPS) against a few headline models from the official leaderboard: the gap to Timestral-209M as one big number, a ranked bar list, and where the king's geomean would place among the leaderboard's no-leak entries.
- Five tiles. Lineage tokens (king lineage; the trainer's counter from
v2, else one contracted leg per promotion generation plus the king's
own), competitions (settlements in
receipts/index.json, with eras and judged challenger legs), miners (distinct hotkeys judged; eligible in the latest settlement), domain experts merged (v2 lane B), and v2 status (live / go-live block + countdown / not announced). - One trend. The king's GIFT-Eval gap to Timestral-209M, settlement by settlement.
Every figure comes from a public object in the manifest bucket; a missing object renders a pending state, never a guessed number.
| Object | Producer | Feeds |
|---|---|---|
receipts/index.json |
validator | competitions, miners, lineage legs |
benchmarks/round-<id>.json and bench_v/round-<id>.json |
trainer and designated main validator (signed reports) | king scores and trend; the page prefers validator entries for matching checkpoint/role |
benchmarks/leaderboard-reference.json |
operator (scripts/refresh_leaderboard_reference.py) |
the reference models |
status/v2.json |
trainer (cascade.shared.v2_status) |
v2 status, lineage tokens (v2), experts merged |
promotions/index.json |
trainer | lineage depth before v2 |
status/chain.json |
validator | era length |
python scripts/publish_website.py --only stakeholdersThis also uploads the versioned GIFT-Eval reference
(cascade/website/data/gift-eval-reference.json →
benchmarks/leaderboard-reference.json). Re-run after any page change.
PYTHONPATH=$PWD python scripts/refresh_leaderboard_reference.py # rewrite the repo file
PYTHONPATH=$PWD python scripts/refresh_leaderboard_reference.py --publish # + upload itThe script downloads every model's per-config results from the Hugging Face
space Salesforce/GIFT-Eval at a pinned revision (≈3 min), recomputes the
space's overall table (gmean of Seasonal-Naive-normalised MASE / CRPS, mean
per-config rank) and re-aggregates the same rows through our benchmark
sidecar's official_aggregate — the aggregation behind every checkpoint's
signed gifteval_mase / gifteval_crps. GIFT-Eval has no zero-MASE config,
so the two readings agree (a unit test pins this): the hero compares like
with like. Commit the rewritten file so each refresh is a reviewable diff.
--models changes the headline set; REF_HEADLINE in the page names the
model the gap is measured to.
Caveats the page states: testnet benches a truncated battery
([eval] cascade_bench_max_series), so its gap is labelled not comparable;
the leaderboard sorts by mean per-config rank, which needs per-config rows
our signed report does not carry, so the king is placed by geomean.
The trainer also publishes warm_start directly from the current era's
warm_start_ckpt, before the first settlement. It includes the era start,
generation, member index and size, and identifies the actual merged init
rather than the trained king. The mining page links it at the top and hides
an init from a different era until fresh status arrives.
policy exposes selected values from the running trainer's resolved config:
training budgets, CPU generation limits, expert-selection/acceptance limits
and the pinned MarketCast base. These fields are unsigned discovery data,
not a replacement for the enforced contract or signed receipts.
cascade status --json re-fetches the chain, v2 and all lane queue feeds,
reports timestamps/availability, and exits 3 if required live feeds are stale
or missing. Unknown policy fields are retained for agents. The new fields
need the updated trainer; they are not inferred from a miner's local config.
scripts/publish_website.py --dry-run includes llms.txt; publishing the
site also uploads it as plain text. Both pages advertise the guide. Static
Vercel deployments serve the packaged file with a short cache lifetime.
The guide describes read-only discovery and requires human authorization
for submissions, reveals and paid actions; publishing a guide does not
change those CLI safeguards.
Unsigned and presentational, published by the trainer from
RollingScheduler.tick (rate-limited to one upload per 2 min, with a fresh
timestamp even for unchanged state; also before v2 is live, when it carries golive.state = "unannounced" for the [v2] from_block = 0 placeholder or "scheduled").
Schema 1:
golive {from_block, state, blocks_to_go};block,as_of,netuid;model— the MoE preset's card (name, total / active params, experts, top-k, MoE blocks);king {era, pointer, routing_health {experts, bins, layers[{layer, tokens, max_over_mean, cv, dead, entropy, heat[32]}]}, health_source}— the current king's per-expert load summed over the domains' routing probes;lane_b {active, from_block, domains, targets, horizon_targets, max_targets_per_leg, payout {unit, full_pay_tiers, half_life, floor, ttl, share, slots, remainder}, verdict {...}, gate {open, opened_block, reason, lineage_tokens, open_tokens, routing_overlap, routing_jaccard, latest, history[{era, block, lineage_tokens, overlap[], jaccard[]}]}, legs {...}, per_domain {target: {pending, open, finished, pending_merge, selection, dropped}}, merges[], merged_total}—targetsis every DEC-CA-0059 emission slot (the domains, thenhorizon:<h>per[eval] scored_horizonsrung: 10 slots on mainnet);payout.slotscounts targets;from_blockis B's independent era-start launch (0 = off), not the v2 lane-A launch.acceptance_from_blockis the separate acceptance-rule activation; pending merges are qualifications, andoutcomesin the trainer's lane-B status distinguishesacceptedfromacceptance_rejected(an accepted row is not a dropped leg);lane_a {round_id, epoch_start_block, ladder {king_share, king_share_bp, prev_bp, event, crown_lcb, crown_units, crown_tier, crown_start_bp, court_hotkey, court_until_block, slashed, slash_reason, slash_until, key_lapses, funded_by, former_share, king_slash, lane_a_cap, prelaunch_burn}, king_check {result, round_id, block, lcb, margin, king_pointer, king_hotkey, reasons}}— the DEC-CA-0055 records of the latest receipt (present once a receipt carries them). A crowning's start share is margin-tiered (owner 2026-10-20):[v2] ladder_start_tiers([[0.50, 8000], [0.25, 6500], [0.10, 5000], [0.01, 2500]], placeholders) on the crowned challenger's verdict LCB when it is in increment units, else the fixed[v2] ladder_start_level_bp(5000); thecrown_*fields are null on every other settlement. The audit's weights check re-derives the units, the tier and the start from the verdict (a forgedcrown_lcb, tier or start share fails it). Calibrate the tiers on seed replicas before arming (the 0.50 tier must be rare under noise); keep the lowest threshold at[v2] win_margin_end. A "crowning LCB below the lowest ladder_start_tiers threshold" warning in the validator log means the two disagree. The key slash escalates per reign (owner 2026-10-20): the first operator-paid lapse holds 1 % for at least[v2] key_slash_min_eraseras (6), then until recovery is recorded under a working rotated key; a seed-epoch carry can record recovery without retraining. A second lapse holds it until the reign ends;operator_faultnever counts. Validators derive it from the manifests'funded_by(ChampionState.king_key_lapses / king_lapse_open / king_slash_until_block); the audit replays it from the ladder record'sprev_*fields, satellites bind those to the previousstate_post. The trainer keeps the failed-key digests inera_state.json(king_failed_keys): deleting that file (or the key) lets the same failed key lift the slash — don't. The "key fixed" probe is auth-only (no Lium balance read exists);lane_c {active, from_block, state, share, takes_from, targets, payout, verdict, legs, per_target {t: {holder, queued, open, finished, leg_slots, pending[]}}, holders_source, trainer}— only once[lane_c] from_blockis typed (absent ⇒ the pages read "not live"). A holder carries its win'stier/full_eras/below_floor.- Gain-tiered full-pay period (owner 2026-10-20).
[lane_b]/[lane_c] full_pay_tiers(overlay-settable) and[marketcast] full_pay_tiers(overlay-settable) replacefull_rounds(a config still typing it is refused at load):[[lcb_threshold, full_eras], ...], default[[0.05, 10], [0.02, 5], [0.01, 2]], thresholds strictly descending in (0, 1),full_erasnon-increasing and<= win_ttl_eras(lane C's TTL for[marketcast]). A win pays in full for thefull_erasof the highest threshold its gain LCB meets, then the unchanged decay and TTL; the tier is fixed at the win and stamped on everylane_b_pay/lane_c_paywin (gain_lcb,tier,full_eras,below_floor) and on thestate_postslot. The defaults are UNCALIBRATED placeholders — lane-B domain LCBs are noisy (few windows and sources per domain): before arming, set the thresholds from real win LCBs (e.g. the testnet receipts' passingtarget_lcb/ confirmedgain_lcb) and keep the lowest threshold atgain_margin(a validator log line "BELOW the lowest tier" means a win clearedgain_marginunder the lowest threshold). A tier change applies to wins made after it only.
The trainer holds no validator slot state, so the lane-C holders and the
lane-A ladder are read off the latest receipt (the anchor's
receipts/<validator_hotkey>/latest.json, else the shared pointer):
lane_c_pay.wins are the paid live slots, enriched with the holder's
confirmed row; rows with outcome pending / deferred are the pending
confirmations (holders_source = "receipt"). An expired or deregistered
holder's slot burns and shows no holder. The read is unverified: the doc is
presentational, the pages read the signed receipt themselves.
The mining dashboard's lane tabs and v2 strip read it with the signed
receipts (lane_b, lane_b_pay, lane_c_pay, emission, king_check,
weights) and, per settlement, the receipt index row's lane_b_verdicts,
lane_b_pay, lane_c_verdicts, lane_c_pay, emission_split (lane C
included, never counted as burn), ladder and king_check; the lane-C page
also reads the trainer's status/lane_c.json (queue, open legs, outcomes).
tests/unit/test_website_lanes.py checks that the fields the page reads are
fields these writers emit. Verify:
curl <endpoint>/<bucket>/status/v2.json | jq '.golive, .lane_a, .lane_c.per_target'.
cascade round (the miner CLI, cascade/miner/dashboard.py) prints the
same lane rows from the receipt index: the lane-A ladder share / court / key
slash / king check, lane-B and lane-C verdict counts and paid slots, and the
emission split.
For shard-mode A/B legs, CPU preparation finishes before GPU rental. Miners
can inspect generation_progress in cascade status --json, or the matched
submissions[].generation report with --hotkey and --ref. It reports CPU
capacity/boot waits, generation, hashing, upload and the parent-verified ready
state, with generated points, series, rate, workers and stored/uploaded bytes.
CPU counters refresh at [generation] progress_seconds (15 by default); public
status publication adds its own interval. Each report has its own as_of.
A fresh status feed does not make an old generation report fresh.
[wandb] enabled = true also opens a separate job_type=generation run,
tagged phase:generation and hotkey:<address>. The orchestrator logs
generation/* counters, stage, final CPU seconds and stop reason. A retry
resumes the source/seed job's run; its elapsed-time counter can restart. This
run is shared when A/B reuse a corpus and does not claim a lane-specific
training result. Independent W&B runs use reinit="create_new" so a second
generation or trainer event cannot close the first. Use the locked W&B extra
(version 0.28 or newer) and configure a public project for
miner access. W&B outages do not fail generation. The CPU VM gets only a
scoped progress-object PUT link; no W&B or corpus-store account key is sent
to the generator. Final corpus metadata remains in meta.json even with
W&B disabled. Reports without a queue-attempt binding are explicitly marked
generator_ref_only; neither counters nor a completed upload prove a reward.
Private storage defaults to [storage] manifest_bucket on the primary S3
service. To use R2, pre-create a private bucket and set these fields within
[generation] (the checked-in mainnet config still uses primary):
storage_backend = "r2"
bucket = "cascade-private-corpora"
s3_endpoint = "https://<account-id>.r2.cloudflarestorage.com"
s3_region = "auto"
access_key_env = "CORPUS_S3_ACCESS_KEY"
secret_key_env = "CORPUS_S3_SECRET_KEY"
shard_encoding = "float32_bss"
compression_level = 3
progress_seconds = 15Supply those two environment variables to the orchestrator using the normal secret-management path. Values never belong in TOML or CLI arguments. R2 requires an explicit HTTPS endpoint and bucket; a bad configuration fails instead of silently selecting another account. Other manifests, receipts and checkpoints keep their existing storage configuration. Select the corpus backend before starting work. Changing a live corpus bucket requires a drained queue or a verified migration of its index and objects; the setting does not migrate existing corpora or refresh already dispatched links.
Objects live under corpora/round-<seed-and-source-id>/<hotkey>/: Parquet
shards, corpus.json, meta.json and the latest progress.json.
corpora/index.json holds per-corpus identities and observed settlement
outcomes. The path's round identifier is a generation-seed/source identity,
not necessarily the settlement round number. GPU workers receive scoped GET
links, verify shard hashes and remove downloaded shards after use. CPU VMs
currently retain the full local corpus through the final digest pass, then
upload shards and the index last. Size CPU disk for the whole corpus:
50B float32 points are 200 GB before compression, plus working space.
Compression savings depend on the generator; allow for poorly compressible
data. Provisioning, hashing and uploading are outside the 90-minute window.
A GPU worker that streams its corpus retries each shard fetch: a transport
error, a short body (fewer bytes than Content-Length) or a sha256 mismatch
is retried up to 5 tries with exponential backoff and jitter (2 s doubling,
capped at 60 s). The partial file is deleted between tries. This shard loop
is the only retry layer; transfer.get_file runs once per try. A 401/403 on
a link past its signed expiry raises ShardLinkExpired straight away,
because retrying a dead link cannot work. The orchestrator has to re-sign it.
A retry fetches the same object and only verified bytes are read, so the
series that train and their order do not change. These retries ship with
the next worker image. Images up to rc15 still fail on the first bad
download (2026-11-02: c4-00030.parquet fails its sha256 at step
7650/23841 on an 8xH200).
[round] funded_prefetch_shards = true (the default) covers older images.
When a shards leg has urls.json, the dispatcher pushes prefetch.sh and
prefetch.tsv next to the corpus and starts the prefetch in its own session
beside the worker. The prefetch does not delay the launch. It downloads every
shard in the order the reader consumes them, 8 at a time, using curl with
timeouts and 5 tries each. Links reach curl on stdin and are never logged.
Each shard is checked against corpus.json and written to a hidden temp
file, then renamed into place. The reader opens only <name> and reads a
shard that is already on disk instead of downloading it, so it never sees a
partial file. The prefetch is killed when the worker exits, and it stops by
itself once the leg's shell is gone. A reattach never starts it again.
Its log lines go to the leg's stderr with the prefix cascade-prefetch:.
The prefetch never fails a leg. It skips with a WARNING when free disk is
below stored_bytes + 10% + 20 GiB at start. Before every download it
checks free disk again and stops once less than 20 GiB is free, because
other lanes on a multi-lane pod share the volume. When a shard never
verifies, it gives up on that shard with a WARNING and the worker fetches
that shard itself, as before. On an expired link it stops. The prefetch
records its pid in the leg's run dir, so a detached timeout or cancel kills
it along with the worker. A fresh launch first kills any leftover prefetch
of the same corpus.
A worker that still dies fetching shards is classed as an operator fault:
it is requeued and does not burn the miner's attempt. This applies only to
shards legs, where no miner code runs on the pod. It also applies only when
the failure's final traceback exception is the shard error itself. On a
streaming leg the generator could print the same text, so there the exit
stays infra. The dispatcher parses that exception line from the worker's
stderr itself. A traceback marker printed into the log tail is never trusted.
The corpus is reused across retries, so a stored object that is persistently
bad would fail every retry. To bound that, a corpus (hotkey, seed, ref) gets
at most 2 unburned shard faults per trainer process. After that the leg is
classed infra and burns an attempt, as before. Check the stored object
when the log shows shard fetch failure #3.
A detached kill (timeout or cancel) kills the leg's whole session. The
guarded worker runs in its own process group inside that session, so a
group kill would miss it. The kill also checks /proc/<pid>/cmdline for
each recorded pid first, so a pid the kernel has reused for another process
is never signalled.
For scale, a 50B-point float32_bss corpus is about
46.5 GB in about 3,000 shards of roughly 15.5 MB each, so the check needs
about 72 GB free. Set the knob to false to go back to streaming only.
Mainnet explicitly selects float32_bss: schema 3 stores typed float32 lists
with Parquet byte-stream split and Zstandard. It preserves the exact stored
float32 values, optional group labels and logical corpus digest; file hashes
and Parquet bytes change. Legacy encoding remains the default for absent
settings and preserves schema-1/2 bytes, including groups=None. Neither
format changes model-source pins or the training contract digest. CPU and GPU
reader images must include schema-3 support before enabling it; old images
must not receive these corpora. Qualify upload, readback, digest and recovery
against the actual private R2 bucket before launch.
Periodic upkeep verifies the configured main validator's anchored receipt
before recording an A settlement or publishing a king corpus. The default
publishes its recipe and four sample shards to the primary public store,
never by making the R2 bucket public. An observed king is durably marked
before copying, so a failed public upload cannot make it eligible for expiry.
Completed non-kings become eligible after retention_days = 14; current
queued/in-flight entries, crowned corpora and unresolved/unmatched outcomes
are retained. Upkeep consumes the latest anchored receipt, so a missed older
settlement can remain unresolved and retained. B/C-only corpora without a
matched A receipt also remain retained. Unindexed partial uploads require
operator reconciliation; this is not an orphan-object sweeper. Invalid or
unavailable receipts hold maintenance rather than authorizing deletion.
Observability only ([wandb]; numbers never feed scoring or weights). One run
per leg, tagged so a public project filters cleanly:
- every run:
round:<id>,hotkey:<hk>,uid:<n>,size:<preset>,role:<log role>andlane:A|B|C; lane-B / lane-C legs addtarget:<t>(a DEC-CA-0059 multi-target leg's targets comma-joined, e.g.target:energy,horizon:64). The run config carrieslaneandtarget; a generic lane-Cfinetunerun would also carry the recipe (recipe_lr_scale,recipe_steps_fraction,recipe_warmup_fraction,recipe_schedule), but lane C is MarketCast-only today and themkt_finetuneworker path (cascade/trainer/worker.py,marketcast_leg.py) opens no W&B run, so lane-C legs log nothing. Expert and fine-tune legs keep thechallenger-<size>role (the receipt role,remote.receipt_role, is unchanged) — filter onlane:. - run ids are deterministic in
(round, role, hotkey)plus, off lane A,(lane, target)(wandb_sink.wandb_run_id): a retried leg resumes its run, and the same hotkey's lane-A, lane-B and lane-C legs in one round never share one. - trainer events (short runs,
job_type= the event, tagevent:<kind>):king_check— each DEC-CA-0055 king check the trainer reads off the receipts (result, LCB, margin, king pointer, ladder share, reasons; once per check);promotion— each published promotion record (generation, members, effective era). - validator summary (
[wandb] validator_summary = true, off by default, independent ofenabled): onesettlementevent per published receipt — lane-A verdict, lane-B / lane-C verdict counts and paid slots, emission split, ladder and king check. Logged after the weights and the receipt are out; no key, no package or a W&B outage is a silent no-op. Validators never need W&B.
Funded legs run on the payer's isolated
pod, which receives no orchestrator environment: W&B reaches it only through
[wandb] funded_key_env, the NAME of an orchestrator env var holding a
SEPARATE key scoped to the public project, forwarded to the pod as
WANDB_API_KEY (the operator's own WANDB_API_KEY is refused). Mainnet
names WANDB_API_KEY_FUNDED: export it on the orchestrator (an unset
variable is skipped silently). The testnet template sets no
funded_key_env, so testnet funded legs log nothing. The payer can read the
key from their pod.
The hero and trend read both benchmarks/round-<id>.json and
bench_v/round-<id>.json, preferring the validator entry for the same
checkpoint/role. Under validator-source A rules, the trainer report can contain
only the init while the main validator report supplies king/challenger scores.
If figures stay missing while settlements land:
curl -sI <endpoint>/<bucket>/benchmarks/round-<id>.json— a 403 means the doc is there but private (backfill the canned ACL withscripts/make_bench_public.sh), a 404 means the bench never ran or never published — check the responsible trainer/main-validator log. Check both report prefixes; a missing trainer king row is expected under validator source.- On the Vercel origin,
curl -sI https://<domain>/benchmarks/round-<id>.json— a 404 there with a 200 at the bucket means the corresponding rewrite (/benchmarks/:path*or/bench_v/:path*) is missing fromcascade/website/vercel.json. - Only a report carrying a
role = "king"entry with all six numbers moves the hero.
To backfill a report by hand: scripts/publish_round_bench.py (see its
--help; reports are immutable once published).
The dashboard's Benchmarks tab shows GIFT-Eval, BOOM, TIME and FEV
(CRPS and MASE). Older reports without FEV show --; FEV does not change
the existing six-score promotion geomean.
Its fixed comparison target is Timestral-209M, loaded from
benchmarks/reference-timestral-209m.json. Score the official pinned
checkpoint with the same sidecar, data revisions and full battery as the
king (--suites gift-eval,boom,time,fev), then prepare the document with:
PYTHONPATH=$PWD python scripts/publish_reference_bench.py /out/report.json \
--preset timestral-209m \
--source 'mistralai/Timestral-209M@<40-character-revision>' --dry-runRemove --dry-run to publish after reviewing the reference. This preset
requires all eight measured losses and a pinned model revision. Until the
document exists, the page shows the reference as pending. Do not substitute
the model card's FEV skill scores for the sidecar's losses. The stakeholder
scoreboard separately compares GIFT-Eval against the refreshed Timestral-209M
leaderboard entry using the sidecar's aggregation.
The Benchmarks chart compares models on a Pareto frontier. Select a benchmark and Size × MASE, Size × CRPS or MASE × CRPS. Size is total checkpoint parameters; active MoE parameters appear in point details. Coral arrows follow the last scored checkpoint of each king's reign in settlement order. Missing scores/sizes and known data-revision mismatches with the current king are omitted rather than estimated. Records without data revisions are historical comparisons, not verified same-data runs.
data/benchmark-models.json supplies the packaged model catalog; the page
prefers benchmarks/model-frontier.json, uploaded by publish_website.py.
The initial catalog has published GIFT-Eval losses using Cascade's existing
97-configuration aggregation. Those external models were not rerun by Cascade.
BOOM/TIME/FEV references require measured sidecar results. To merge one:
PYTHONPATH=$PWD python scripts/publish_reference_bench.py /out/report.json \
--preset timestral-209m \
--source 'mistralai/Timestral-209M@<40-character-revision>' \
--dry-run --out /tmp/timestral-reference.json
PYTHONPATH=$PWD python scripts/refresh_pareto_reference.py \
--reference-doc /tmp/timestral-reference.jsonThe refresh fetches public leaderboard/checkpoint metadata and counts Cascade preset parameters on a meta device; it runs no model inference and publishes nothing. Review the generated catalog before the normal website publication.
- Top bar shows LIVE; the hero names Timestral-209M with a gap and a ranked bar list including "Cascade king".
- Five tiles: lineage tokens, competitions, miners, experts merged, v2. Before v2 the last two read "v2 domain experts" / "go-live not announced" (or the go-live block with a countdown).
- The trend has one point per benched settlement.
- Dark mode and phone width: tiles fall to 3 then 2 columns; the hero stacks.
- Three blocks, five tiles, one chart; labels and tooltips, no prose.
- No invented numbers: reference figures come only from the refresh script, king figures only from signed bench reports; a missing source pends.
- The king's number and the reference must share an aggregation; if that ever stops holding, label it on the page.
- Detail (verdicts, cohort, lane B, MoE health, training curves) belongs on
the mining dashboard (
index.html), not here.
Lane C can run several fine-tune competitions at once. Each one is a
[lane_c.competitions.<kind>] table in chain.toml (weight,
from_block, until_block, wind_down, wind_down_max_eras). The shipped
config has one: mkt (MarketCast), weight 1, no end.
Add a competition (a release). A competition kind is code: its eval must be forward-only (it scores only windows that close after the entry is trained).
- Register the kind with its eval in
cascade/shared/config/lane_c.py(register_competition_kind(kind, eval=..., forward_only=True, from_block_table=..., parse_value=...)). Setfrom_block_tablewhen the eval has its own go-live table (MarketCast:"marketcast"); otherwise the start is the competition table's ownfrom_block. - Ship its judging pipeline (validator), its trainer leg and its audit replay with the same release.
- In
chain.tomladd its targets to[lane_c] targetsand add[lane_c.competitions.<kind>]withweightand an ERA-BOUNDARYfrom_blockat or after[lane_c] from_block. The loader refuses a non-boundary block, an unknown or non-forward-only kind, a target whose kind has no table, and a negative weight. - Every validator and the trainer must run the release before activation. Weights are normalized over every configured competition, including pending competitions, so the new share is reserved once that configuration applies. Until the competition starts, its part burns. Tell miners the new per-target share before you ship.
Re-weight or end a competition (an overlay or a release). weight and
until_block of a configured competition are overlay-settable
(DEC-CA-0057). An overlay replaces the whole [lane_c] competitions table,
so restate every competition with all its keys:
[lane_c.competitions.mkt]
weight = 1.0
until_block = <era boundary>
wind_down = "finish_full_pay"
wind_down_max_eras = 6until_block must be an era boundary after the start, and the overlay's
apply block must be before it (an end is never moved to or from a block
already passed). The settlement AT until_block is the last one judged.
From that block the intake answers 409 competition_ended; the trainer
fails a queued entry whose leg ([lane_c] leg_hours) cannot finish before
it, as competition_ended (uncharged). Winners wind down per wind_down.
An overlay cannot add or remove a competition, change from_block or the
wind-down rule; those need a release. The receipt's lane_c_pay records
each competition's share and state, and the audit replays it.
The owner changes [scoring], [eval], the [round] schedule keys
(epoch_blocks, epoch_blocks_prev, epoch_activation_block,
rolling_from_block, era_settlements, round_hours, blocked_hotkeys,
blocked_from_block, commit_floor_block, funded_field_cap), [v2] runtime policy
(never [v2.training]), [lane_b], [lane_c] and the listed numeric
[marketcast] controls by publishing a signed overlay. Validators adopt
it by stake. Everything outside the allow-list is still a release.
Install the release supporting these controls on the fleet once. After that, supported tuning needs no validator restart or code update. Inspect the installed release's exact policy locally (no wallet or network):
cascade-validator --netconfig-policy| Control | Update path |
|---|---|
| Existing scoring rules' thresholds, eval sample counts, lane shares and reward tiers | Signed overlay |
| Existing lane-A benchmark source, rule activation, report timeout and sweep budgets | Signed overlay; source uses the already pinned trainer/main-validator identities |
| MarketCast gain/confidence/calibration thresholds, minimum evidence, reward tiers and report/infra limits | Signed overlay; exact keys in --netconfig-policy |
| CPU generation time, resources and corpus target | Operator config; no other validator update |
| Trainer-declared optimizer/recipe/budget terms | Trainer config and signed manifests; locked terms remain release-only |
| New model/scoring code, locked training terms, trusted keys, storage locations, MarketCast base/universe/data contract | Release |
Every overlay still passes the normal loader and cross-field checks, is owner-signed, requires the configured permit-stake lock-in (51% in the shipped config), and takes effect at an era boundary. Historical settlements keep their original configuration. Changing a parameter does not implement an algorithm a validator's code does not already support.
One-time setup (a release): set [netconfig] owner_hotkey in
chain.toml to the owner's MAIN validator hotkey and ship it to every
validator and the trainer. [activation] feature must stay set: adoption
rides the readiness note. Validators need read access on
config/network/ in the manifest bucket. Publishing needs write access on
that prefix (the trainer's Hippius S3 pair works).
Each change:
-
Write the FULL set of overrides as a TOML file. Overlays are snapshots, not deltas: seq N+1 replaces seq N, so restate every key you want to keep. For example:
[scoring] min_windows = 250 [lane_b] share = 0.45
-
Pick
effective_block: an era start comfortably ahead of the head. If the 51 % lock-in lands after it, the overlay applies at the first era start after the lock-in, and every changed*_from_blockin it must then still be at or after that block, or the fleet refuses it. -
Dry run (the default; signs and writes nothing):
cascade-validator --netconfig-overlay overlay.toml \ --netconfig-effective-block <era start>It prints the next
seq/prev, the digest and itscfg-<8 hex>note segment, the merged diff againstchain.toml, and every check a validator runs (allow-list, merged load checks, era start, retroactivity), thenVALIDorINVALID. -
Publish with the owner wallet:
cascade-validator --netconfig-overlay overlay.toml \ --netconfig-effective-block <era start> --netconfig-publish \ --wallet-name <owner> --wallet-hotkey <main validator hotkey>It refuses a wallet whose hotkey is not
[netconfig] owner_hotkey. It writesconfig/network/seq-<n>.json, thenconfig/network/latest.json. Nothing is written on chain: your validator signals thecfg-segment in its own note on its next poll. -
Watch the logs for
netconfig: overlay seq N … verified, thenADOPTED — in force from block B(orREFUSED, with the reason; fix it and publish the next seq). Receipts from block B on carrynetconfig_seq/netconfig_digest.
The trainer runs the same resolver read-only (netconfig_state.json in its
work root). It never signals, and it switches its rolling scheduler and
promotion engine to the overlay at the same block.
The release presets require exactly 32 channels in every synthetic yield, including
optional synthetic generators supplied with MarketCast recipes. The private eval
pool accepts native widths up to 32; narrower real data stays valid. Verify the source with the new CLI before
uploading; --skip-runtime does not check output width. cascade init creates an
32-channel starter. Stream parents, CPU corpus generation and GPU shard readers
check widths; incompatible cached corpora are regenerated before GPU rental.
Before activating v2, verify the chosen initial king/baseline generator too. An inherited v1 source producing fewer than 32 channels does not satisfy the v2 policy; prepare a compliant baseline source/ref before launch. Archived source is kept unchanged for replay, and historical pre-v2 training resolves the original floor of one channel. Do not pad or duplicate archived data silently to pass the gate.
Run the launch qualification against the immutable source that the initial king will actually use, on a host with the production generator sandbox:
cascade-trainer --chain-toml chain.toml --work-root ./_train_work --preflight-v2-baseline NAMESPACE/GENERATOR@sha256:DIGESTThis fetches the pinned tree, runs full bounded verification, and records a
local proof tied to its ref, contract, generator policy and verifier source.
It rents nothing. The first v2 king leg refuses rental without a matching
proof and a pinned validator receipt anchor. A launch root survives a later
v2 dethrone; changing the verifier or policy requires qualification again.
The trainer and validator may share the configured signer; receipts then share
that trust anchor rather than providing an independent attestation. The benchmark
owner is pinned to the main validator in the mainnet template. Do not assume
installation has qualified the baseline or completed release configuration.
status/v2.json publishes launch_preflight=blocked with its reason, and
the CLI does not report training readiness while that blocker is present.
A fired promotion is not sufficient to change the training init: it must have
a timely published settlement notice. A record that cannot be published at
its fire boundary is retracted (generation N RETRACTED) and the passing
check re-read at the next boundary. If a notice still missed its window, the
generation is dead for every node: the trainer keeps the prior verified
generation and logs promotion N not installed: … announced too late, every
validator skips that generation (never effective … skipped), and the next
passing king check fires the following generation. A promotion record publish REFUSED (another body already published for that generation) needs operator
reconciliation. If the bound evidence is unreadable, scheduling holds until
it can be read. Do not backdate a record or edit the era ledger to force it
through. Verify the notice/report replica and restart paths during the
integration qualification.
Correction (DEC-CA-0067): the mainnet chain.toml's eval_n_windows = 4000
sits under [round], where nothing reads it. It is a dead key: v2 settlements
judge on [eval] n_windows (2000) until the DEC-CA-0067 switch. From the
switch era, the bundled [v2] eval_n_windows_after = 4000 is what moves the v2
verdict budget to 4000. Do not move the dead key into [v2] by hand, because
that would change the budget on one node only and fork it. [v2] eval_n_windows
(parsed, 0 = off), when set, is applied only to settlements judging v2 eras.
Trainer, validators and audit must share that configuration. The legacy single-horizon draw's 1,200-window target is not
applied to that v2 budget. New pool builds default to 800/domain,
400/domain-frequency, 32 maximum native channels and source-family clustering.
Update any explicit 600/300 publish-cron flags; repository edits do not change
an installed cron or publish a new snapshot. Rebuild from the existing mirror,
inspect family/drop/window counts, and publish only for a future settlement.
See pool build rules.
A new 32-channel compliant baseline and GPU qualification remain required.