Skip to content

About

Evolutionary program search for agents that learn world models and plan in partially observed dynamic mazes. ShinkaEvolve jointly evolves predictive representations, online learning rules, and planners, with controlled experiments on prediction, decision-making, and generalization.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

ShinkaDreamer: Joint Evolution of Map Memory and Planning in a Partially Observed Dynamic Maze

Scientific implementation report — original-proposal track, with a frozen-program assessment on 256 fresh paired cases.

Experiments are stopped for an accountability review. The broader study is incomplete; no calibration or further search is running. The contribution record connects calls to executable changes and measured results. A separate supervision audit records usage omitted from experimental call totals. Paper synthesis follows completion of the study.

Abstract

We implement the original Sakana ShinkaDreamer proposal: evolve an executable agent whose world-model updater integrates partial observations and whose planner uses that internal map to navigate a dynamic maze. Selection combines task reward and current-map reconstruction accuracy with the proposed weights of 0.6 and 0.4. The implementation preserves the 15×15 maze,5×5 observation window, keys, gating door, moving enemies and changing walls, with explicit repairs to the example code's coordinates, mechanics and evaluator interface. In an initial development characterization, the unevolved seed completed 32 valid episodes and escaped in none;20 ended in capture and 12 timed out. Mean map accuracy was 98.31%, with 32.90% mean final map coverage. These results characterize the starting agent and show that accurate remembered cells alone do not establish successful navigation. The first native ShinkaEvolve descendant escaped all five search cases. A subsequent paired development check of generation 2 found 32/32 escapes versus 0/32 for the seed, including all 27 cases outside the five-case search panel. Mean fitness rose from 0.4967 to 0.9353 and final map coverage from 32.90% to 83.64%; reported-cell accuracy decreased slightly to 98.04%. On a subsequent disjoint 64-case selection-validation panel, both descendants escaped 61/64 cases, versus 0/64 for the seed. The prespecified fitness rule selected generation 2, but its advantage over generation 1 was unresolved. Its fitness gain over the seed was 0.4171 (descriptive paired 95% interval 0.3817–0.4476). A final assessment of the frozen generation 2 on 256 fresh paired cases found 239/256 escapes (93.36%) versus 2/256 (0.78%) for the original seed. Mean fitness improved by 0.4026 (paired 95% interval 0.3835–0.4201), almost entirely through task reward; the change in reported-cell accuracy was unresolved. All 512 assessment episodes took 64.06 seconds with no experiment-model calls. This establishes improvement over the original weak seed on the declared task distribution. A subsequent 256-case control experiment found source-specific benefits from persistent spatial maps and the hardcoded future-risk heuristic; fitness superiority over a competent handcrafted pathfinder remains unresolved. Independent discovery reliability is still under study.

1. Introduction

Partial observation makes memory and planning central to maze navigation. An agent must retain information about previously observed passages and objectives, reconcile that memory with a changing world, and choose actions using incomplete knowledge. The original Sakana proposal addresses this problem by jointly evolving two Python functions:

world_model_step(memory, local_obs, last_action) -> memory
planner(memory, local_obs) -> action

The scientific question is whether program evolution improves task performance and internal-map quality through interpretable changes to these functions. Possible changes include pathfinding, uncertainty, remembered enemy sightings, exploration and replanning. Helpers and memory structures remain evolvable; this is not restricted to scalar tuning or a fixed algorithm catalog.

Here, world model means the agent's maintained representation of the current world. Future prediction and learning an unknown transition law are not required by this experiment. ShinkaDreamer is not an implementation of neural Dreamer. Earlier predictive-learning extensions are retained in a separate historical archive; none of their measurements or figures is presented as evidence for this objective.

2. Methods

2.1 Environment and information boundary

The environment contains a 15×15 maze, two keys, a locked door gating the exit and three anonymous enemies. The agent receives a 5×5 local observation. It chooses one of nine attempted displacements, including waiting, and an interaction flag. Enemies also draw uniform attempted displacements; blocked moves remain stays. Walls change every 25 transitions and episodes end on capture, escape or the 200-transition horizon.

The implementation reuses the existing repaired maze. Border walls remain fixed, layouts have feasible objective routes, diagonal corner cutting is disallowed, and wall closures skip occupied cells. Interaction opens an adjacent cardinal door before movement when both keys are held and collects a key at the movement destination. Contact with an enemy before or after enemy movement is fatal. These are explicit interpretations and repairs of the proposal's prose and example, documented in implementation details.

Candidates receive local observations, inventory, door state and observable movement/interaction feedback. The hidden world, world seeds and scoring remain inside the evaluator. The candidate runs in the existing restricted Python process, with independent candidate randomness. No forecast interface, hidden movement-law condition or prediction-learning intervention is imposed.

2.2 Initial program and joint evolution

The initial program closely follows the proposal: integrate observations into a persistent map, retain enemy sightings for five steps, move greedily toward the nearest remembered objective, and otherwise wander randomly. It has no engineered pathfinder or learned dynamics predictor. Repairs apply actual displacement before map integration, maintain observable inventory and door state, represent anonymous sightings, and permit interaction on a key already under the agent.

The native launcher uses upstream ShinkaEvolve revision 9912af1. The proposal's nominal configuration has 100 total slots, four islands and five development episodes per evaluated program. We implement it in separately reviewed blocks. The first block has a six-slot ceiling, including the seed and failures, 30 actual provider calls across all roles, 1,200 summed provider seconds and a 40-minute invocation ceiling. It does not automatically continue toward 100 slots.

The bounded launcher retains native weighted parent selection, archive/top inspirations, migration, diff/full/crossover operators, reward-only model allocation, real local BGE embeddings, conditional novelty judging, summaries/recommendations and prompt coevolution. Mutation allocation uses Astra/high and Sol 6.1/high through the strict subscription route; all auxiliary roles use that same route, and embeddings run locally. Configured mechanisms and actual events are reported separately. See the frozen protocol, staged plan and source-level Shinka study.

2.3 Original fitness

For an episode with length $T$, collected-key count $K$, door indicator $D$, escape indicator $E$ and capture indicator $C$, the original task reward is

$$S = \mathrm{clip}_{[0,1]}\!\left(\frac{0.1T+20K+30D+100E-50C+50}{250}\right).$$

After each observation and planning call, but before applying its action, the evaluator compares the reported believed_map with current terrain and enemy occupancy. Relative map coordinates are translated privately using the initial agent position. Let $B_t$ contain the reported in-bounds cells at step t. Accuracy is

$$A = \frac{\sum_t\sum_{x\in B_t}\mathbf{1}[\hat m_t(x)=m_t(x)]} {\sum_t |B_t|}, \qquad F=0.6S+0.4A.$$

An empty audit has $A=0$. Fitness is averaged equally over episodes; invalid executions contribute zero and remain in the denominator. This is reconstruction of the present after observation, not prediction of a future outcome. The original positive step reward and reported-cell denominator are retained. Coverage, escape, capture and timeout are reported separately rather than hidden inside the composite score.

3. Initial experiment

Before execution, we fixed 32 development cases, numbered 0–31, and evaluated only the initial program, with a maximum of 6,400 transitions. No fitting, candidate selection or mutation-model calls were performed. These are development cases, not a fresh assessment or evidence of generalization. The first case's replay was selected by index before observing its outcome.

This 32-case characterization is separate from the launcher's five-case selection default. Its score must not be imported as native search fitness; a search must evaluate its seed and descendants on the same configured panel.

The saved measurements, per-episode data and source manifest support all results and figures below. Every figure uses the shared Chromatic Fields style and only this original-proposal experiment's data.

4. Results

4.1 Initial seed characterization

Measurement Initial seed,32 episodes
Escape 0/32 (0%)
Capture 20/32 (62.5%)
Timeout 12/32 (37.5%)
Invalid execution 0/32
Mean task score $S$ 0.172325
Mean map accuracy $A$ 0.983149
Mean combined fitness $F$ 0.496655
Mean final map coverage 32.90%
Mean collected keys 0.53125
Opened door 0/32
Mean episode length 137.06 transitions

Original-proposal initial-agent outcomes, objective components and map coverage

Figure 1. Initial-agent characterization. All 32 episodes are included. Outcome intervals are descriptive 95% Wilson intervals for the development panel; score and accuracy panels retain individual episodes. Coverage is reported independently of accuracy. These are results for one unevolved program, not an evolutionary comparison or a held-out success estimate.

Recorded hidden world and reconstructed map from the first development episode

Figure 2. Actual map memory in the first recorded episode. The first and last recorded pre-action states show evaluator truth alongside the agent's reported map, translated into common coordinates. Hidden truth is a retrospective visualization and was never supplied to the agent. Unknown map cells remain visually distinct. This case ended in capture; the last displayed state precedes the terminal action and is not a post-capture reconstruction.

Execution used 4,386 transitions. Summed measured episode wall time was 2.84 seconds on the current host, including candidate execution; this excludes implementation, tests and figure rendering. That initial characterization used zero experiment-model calls. The subsequent search has its own measurements below.

4.2 First native evolutionary block

The search reused development cases 0–4 for every evaluated program. It reserved four of the six permitted slots: the seed, two valid descendants and one unfinished full-rewrite proposal. Both full-rewrite attempts timed out at the declared 240-second request boundary, triggering the prospective two-failure checkpoint. No valid source or episode result exists for that reserved slot. Three additional database rows are administrative island copies of the seed, not evaluated programs.

Program Episodes Escape Capture / timeout Task $S$ Accuracy $A$ Fitness $F$ Coverage Mean steps
Seed, slot 0 5 0/5 3 / 2 0.236000 0.986904 0.536362 34.31% 150.0
Generation 1 5 5/5 0 / 0 0.908240 0.982192 0.937821 88.89% 70.6
Generation 2 5 5/5 0 / 0 0.910880 0.982193 0.939405 90.58% 77.2
Slot 3, unfinished rewrite 0 — — — — — — —

All 15 executed episodes were valid, with 1,489 transitions. Generation 2 is the search-fitness leader, not a held-out selected champion. Its paired mean fitness gain over the seed is 0.403044: the task term contributes +0.404928 and the accuracy term −0.001884. These gains are attributable to task performance in the objective, not improved reconstruction accuracy.

Generation 2 exceeds generation 1 by only 0.001584 fitness. Almost all of that difference comes from taking 6.6 more steps on average: the original objective rewards elapsed steps. Both escape all five cases, so this ranking does not establish superior navigation efficiency. We retain the proposed reward and disclose this limitation.

Every original-task candidate, objective components and coverage

Figure 3. Search outcomes under the original objective. Episode points and program means use the same five development cases. The unfinished slot has no invented score. Accuracy and coverage remain separate; these are selection data.

Both successful programs are direct seed mutations without inspirations. They separate persistent terrain from transient enemy occupancy, add visit history and frontier exploration, and replace direct target movement with shortest-path planning through keys, door and exit. Their planners use accumulated map memory and fixed danger estimates based on the known uniform enemy movement rule. They also omit some stale dynamic cells from the scored map while retaining them internally for navigation. This selective export and its denominator matter when interpreting accuracy. No learned transition law or isolated causal effect of memory is established by this code inspection.

Actual native lineage, operator events and model-role calls

Figure 4. Actual native search activity. Both mutation models were selected. Real local embeddings and conditional novelty comparisons ran; one comparison exceeded the 0.95 threshold and invoked the subscription-backed judge, which accepted generation 1. Generation 2's similarity was below threshold and required no judge. Neither result is a scientific novelty claim.

The full configuration was enabled, but this short execution did not exercise every mechanism. Diff produced both valid descendants; full rewrite produced two timeouts. No inspiration or crossover was used, no migration occurred, and no summary, insight, recommendation or prompt-mutation provider call ran. The initial prompt archive received native descendant credit; it did not evolve a new prompt. Three programs remain pending in meta memory. At finalization the shared budget gate rejected further dispatch after the two transport failures. We therefore report a partially exercised full-native configuration, not a demonstration of benefits from the complete machinery.

Development escape outcomes and measured search resource use

Figure 5. Outcomes and computational cost. Outcome intervals are nominal 95% Wilson intervals on the development panel; 5/5 gives 56.6–100%. Reusing and selecting on these cases prevents interpreting that interval as held-out generalization evidence. Timed-out requests remain in resource totals; their unreported tokens are unknown.

Recorded first-case replay of the search-fitness leader

Figure 6. Leader replay, development case 0. The case is chosen by index, not outcome. Evaluator truth and the reported map are shown retrospectively; hidden truth was never supplied to the agent. All recorded replay frames and nonwinning candidate sources are preserved in the compact results.

The controller ran for 846.11 seconds (14.10 minutes), excluding implementation and reporting. Five provider requests consumed 818.03 seconds: two successful mutations, two timed-out rewrite attempts and one successful novelty judgment. Reported usage from the three successful requests was 30,416 uncached input plus output tokens and 36,864 cached input tokens. Usage for the two timeouts is unknown; these are incomplete totals, not an estimate of subscription allowance.

The 15 episodes consumed 2.915 evaluator-plus-candidate CPU-seconds, of which 2.110 were candidate CPU-seconds. Separately sampled controller and embedding process CPU were 17.71 and 9.59 seconds; peak sampled combined resident memory was 1.09 GiB. The latter CPU samples are not a complete machine-wide resource audit. Supervising-assistant work is additional and is not converted into a weekly allowance percentage. Native API price estimates are not subscription charges.

The search is checkpointed with workers stopped. Its subsequent development comparison is reported below. The historical assessments remain paused.

4.3 Paired 32-case development comparison

Before execution, the Stage 3 protocol nominated generation 2 because it led both search fitness and task score. Its source, evaluator and limits were frozen. We evaluated it once on cases 0–31 and reused the seed's saved results after verifying all seven evaluator/dependency source hashes. No candidate was edited, no baseline was rerun and no model was called. The five overlapping search cases reproduced every recorded outcome and score exactly. Cases 5–31 are additional development cases, not a fresh or held-out assessment; this panel was already exposed in the seed characterization.

Measurement Saved seed Generation 2 Paired difference
Escape 0/32 32/32 +100 percentage points
Capture / timeout / invalid 20 / 12 / 0 0 / 0 / 0 —
Mean task $S$ 0.172325 0.905300 +0.732975
Mean reconstruction $A$ 0.983149 0.980418 −0.002731
Mean fitness $F$ 0.496655 0.935347 +0.438692
Mean final map coverage 32.90% 83.64% +50.74 percentage points
Mean keys / opened door 0.53125 / 0 of 32 2 / 32 of 32 —
Mean episode length 137.06 63.25 −73.81 transitions

Generation 2 escaped 5/5 reused search cases and 27/27 additional development cases. Its successful episodes lasted 25–130 transitions. The aggregate length difference includes seed failures and is not an escape-speed comparison between successful policies. The fitness gain decomposes into +0.439785 from task reward and −0.001093 from reconstruction. Better task performance therefore persists beyond the five search cases without improved reported-cell accuracy.

Descriptive paired 95% intervals are [0.4137, 0.4623] for the fitness gain, [0.6914, 0.7728] for task score, [−0.0075, 0.0021] for reconstruction, and [43.93, 57.18] percentage points for coverage. These intervals quantify variation across the observed development cases; they do not correct selection bias or establish performance on unseen cases.

Paired original-proposal development effects and outcome uncertainty

Figure 7. Development comparison, all 32 paired episodes. Intervals use 10,000 whole-episode paired bootstrap resamples, preserving the original mean of episode-level reconstruction ratios. The binary bootstrap degenerates when every observed pair has the same escape difference; it cannot establish a perfect population success rate. Separate nominal 95% Wilson intervals are 89.3–100% for generation 2 and 0–10.7% for the seed. These post hoc boundary diagnostics do not remove development exposure or candidate-selection bias.

Every development case, with search reuse distinguished from the additional 27 cases

Figure 8. Every development case. The five search cases and 27 additional development cases are identified explicitly. The comparison preserves all outcomes and uses the same environment case IDs. Reconstruction remains an on-policy current-map measurement under different actions, not a matched-experience prediction experiment.

The new evaluation executed 32 valid episodes and 2,024 transitions, using 5.55 seconds wall time and 5.45 total CPU-seconds, including 4.45 candidate CPU-seconds. No new proposals, mutation calls or auxiliary model calls ran. The seed's CPU usage was not measured in its initial characterization, so no CPU speedup is inferred. Analysis, rendering and supervising-assistant work are additional to these evaluation measurements. All episode data, a paired CSV, uncertainty estimates and rendering provenance are in the Stage 3 results.

Stage 3 is complete. The separately authorized selection-validation step follows.

4.4 Selection-validation and assessment freeze

The Stage 4 protocol fixed the three sources and selection rule before generating 64 private cases. The nomination rule retained generation 2 for best search $F$ and $S$, and generation 1 through the earlier-generation tie-break on escape count. All programs used the same new cases and unchanged evaluator. Pool generation excluded every saved private seed pool and the public seed range below 1,000,000. Public results use ordinal IDs; private seeds remain outside Git. No source was edited or tuned on this panel, and its scores were not imported into the search database.

Measurement Original seed Generation 1 Generation 2, selected
Escape 0/64 61/64 61/64
Capture / timeout / invalid 37 / 27 / 0 3 / 0 / 0 3 / 0 / 0
Mean task $S$ 0.173006 0.867581 0.868744
Mean reconstruction $A$ 0.982144 0.980801 0.981192
Mean fitness $F$ 0.496662 0.912869 0.913723
Mean final map coverage 35.67% 85.15% 83.77%
Mean keys 0.46875 1.96875 1.953125
Opened door 0/64 61/64 61/64
Mean episode length 127.83 59.58 65.61
Evaluator + candidate CPU-seconds 5.146 10.632 11.535

The rule selects the descendant with the highest mean original $F$, then $S$, then earlier generation. Generation 2 is selected, with a paired fitness gain of 0.417061 [0.381668, 0.447600] over the seed. Its escape difference is 95.31 [89.06, 100.00] percentage points; both descendants have nominal Wilson escape intervals of 87.10–98.39%. These descriptive intervals use 10,000 whole-case paired bootstrap draws, with separate-arm Wilson intervals for escape. They do not remove bias from selecting the winner on this panel.

Generation 2's advantage over generation 1 is only 0.000854 [−0.036302, 0.037922] in $F$, with zero observed escape-rate difference and a paired interval of [−7.81, 7.81] percentage points. Superiority is unresolved. The original positive step reward matters again: generation 2 takes 6.03 more steps per episode on average, contributing +0.001448 to the fitness difference, while collecting slightly fewer keys contributes −0.000750. Reconstruction contributes +0.000156. Selection under the requested objective does not establish better navigation efficiency.

Selection outcomes, objective components and all paired contrasts

Figure 9. Selection-validation comparison. All 64 cases contribute to each arm. Shared bootstrap case indices preserve pairing across all programs and metrics; reconstruction retains its mean of within-episode ratios. The selected source is frozen, but this panel remains selection data, not final assessment.

Every selection case, including the six distinct descendant captures

Figure 10. Individual cases and failures. Generation 1 was captured on ordinals 11, 25 and 63; generation 2 on 7, 50 and 62. Neither failed on the other's three capture cases. All failures remain visible and in score denominators. The earlier 32/32 development result therefore does not imply universal escape.

Execution completed 192 valid episodes / 16,193 transitions in 28.18 seconds, using 27.62 total CPU-seconds, including 21.14 candidate CPU-seconds. The per-arm table excludes a small amount of controller/export CPU included in that total. There were zero experiment-model calls, new proposals or candidate edits; no additional Shinka mechanism was exercised in this evaluation stage. Each episode retained the 192 MiB/10 candidate CPU-second limits and had a declared 20-second wall guard. Analysis, rendering and supervising-assistant use are additional to evaluation costs. See the results and paired data.

The assessment freeze bound generation 2, the original seed, evaluator, paired contrasts and 256-case sample before Stage 5. The primary contrast is mean $F$ improvement; 0.05 was specified as a meaningful difference. Development and selection variances gave planning estimates of 0.009 and 0.017 for its 95% half-width. The frozen assessment is reported separately below; its results did not change the selected program.

4.5 Fresh assessment of the frozen program

The Stage 5 protocol bound the earlier scientific freeze to a 30-minute execution/reporting ceiling and 1,200 total evaluator/candidate CPU-seconds. The 256 private cases were drawn after the freeze, excluding all previous private pools, including Stage 4, and the public seed range below 1,000,000. Both unchanged programs ran on every case. There were no new candidates, model calls, sample-size extensions or selection decisions using these results. The assessment pool is now retired from future tuning and fresh confirmation.

Measurement Original seed Frozen generation 2 Paired difference, 95% interval
Escape 2/256 (0.78%) 239/256 (93.36%) +92.58 [89.06, 95.70] percentage points
Capture / timeout / invalid 154 / 100 / 0 16 / 1 / 0 —
Mean fitness $F$ 0.500541 0.903113 +0.402573 [0.383537, 0.420061]
Mean task $S$ 0.179711 0.850380 +0.670669 [0.638845, 0.699953]
Mean reconstruction $A$ 0.981785 0.982214 +0.000429 [−0.001346, 0.002210]
Mean final map coverage 35.72% 79.78% +44.05 [41.16, 46.79] percentage points
Mean keys 0.53125 1.9375 +1.40625 [1.3125, 1.4961]
Opened door 2/256 239/256 +92.58 [89.06, 95.70] percentage points
Mean episode length 133.65 56.03 −77.63 [−86.80, −68.31] transitions

The primary fitness interval lies entirely above the prespecified meaningful difference of 0.05. Its gain decomposes into +0.402401 from task reward and +0.000172 from reconstruction. Reported-cell accuracy is similar, while the evolved program covers substantially more of the map. Accuracy is measured along each policy's own trajectory; this is not a matched-experience prediction test. Episode length includes captures and timeouts and is not a comparison of escape speed conditional on success.

Generation 2 escaped on both cases where the seed escaped, and on 237 additional cases. Separate-arm 95% Wilson escape intervals are 89.62–95.81% for generation 2 and 0.21–2.80% for the seed. All paired intervals use 10,000 whole-case bootstrap draws with the frozen RNG seed, sharing each draw across programs and metrics. Secondary intervals are unadjusted. The zero-width invalid-execution interval reflects no observed invalid runs, not a guarantee of population reliability.

Frozen-program assessment means and paired effects with uncertainty

Figure 11. Fresh assessment effects. All 256 pairs contribute. The orange tick marks the prespecified meaningful difference for the sole primary endpoint, $F$. Arm-level escape intervals use Wilson intervals; paired effects and other means use the paired bootstrap. Reconstruction preserves the equal-episode mean of within-episode ratios. Frozen refers to source code; ordinary map-memory updates remain active within episodes.

All 256 assessment cases and outcomes, including every evolved-program failure

Figure 12. Every assessment case. Generation 2 had 16 captures and one timeout; all remain in the analysis. Ordinal order has no temporal meaning, and private environment seeds are not exported. The original seed's two escapes are highlighted. The dashed line is the mean paired fitness gain.

The complete assessment executed 512 valid episodes / 48,558 transitions in 64.06 seconds wall time and 62.61 total CPU-seconds, including controller overhead. Summed episode evaluator/candidate CPU was 61.33 seconds, the quantity shown in Figure 12. Saved-data analysis, rendering and supervising-assistant usage are outside these measurements. Sources, all public episode records, paired CSV, uncertainty and figure provenance are in the Stage 5 results. Recovery verified these previously completed results without rerunning episodes.

4.6 Competent comparator and mechanism interventions

The next experiment was frozen in commit 7bc6949 before drawing another 256 paired private cases. It compares unchanged generation 2 with the historical handcrafted memory/pathfinding agent and two source-specific interventions. The comparator retains fixed local enemy avoidance; its adapter adds the current-map export and repairs key-underfoot interaction and diagonal closed-door entry. The local-map intervention removes spatial maps outside the current 5×5 view, retaining localization, inventory and visitation history. The no-risk intervention removes the hardcoded next-enemy-move probability calculation while retaining occupied-cell exclusion. Neither intervention disables all memory or isolates learned dynamics. All three controls were checked on the existing 32 development cases before freezing; there were no subsequent outcome-driven edits.

Condition Escapes Capture / timeout / invalid Mean $F$ Mean $A$ Final coverage
Generation 2 243/256 12 / 1 / 0 0.911277 0.981869 80.47%
Handcrafted memory pathfinder 230/256 26 / 0 / 0 0.888316 0.980385 82.35%
Generation 2, current local map only 134/256 18 / 104 / 0 0.800794 1.000000 9.64%
Generation 2, no next-step risk 154/256 102 / 0 / 0 0.743233 0.985432 70.30%

Three primary comparisons were specified: original $F$ against the handcrafted pathfinder, and escape against each intervention. Shared paired bootstrap draws give Bonferroni 98⅓% intervals for approximate 95% coverage of that family. The fitness advantage over the pathfinder is 0.022961 [−0.003703, 0.049810]; the declared primary comparison does not establish superiority or the specified meaningful gain of 0.05. Its escape difference is +5.08 percentage points with an unadjusted secondary 95% interval of [0.39, 9.77].

Access to remembered spatial maps increases escape by 42.58 [34.38, 50.78] percentage points relative to the local-map intervention; using next-step risk increases it by 34.77 [26.95, 42.58] points relative to no risk. Both family-adjusted intervals exclude zero, supported by exact paired binary tests. These are causal contrasts for the specified source changes on this task distribution. They do not show that Shinka reliably discovers these mechanisms or learns enemy dynamics. The local-only map achieves perfect reported-cell accuracy while escaping less often and covering far less terrain, illustrating the objective's denominator limitation.

Fresh competent-control and source-specific intervention effects

Figure 13. Competent control and mechanism assessment. All 1,024 episodes are included. Thick paired intervals are marginal 95%; thin intervals adjust for the three primary endpoints. The orange mark is the 0.05 meaningful fitness gain for the pathfinder comparison. Arm-level escape intervals use Wilson intervals. See the protocol, all paired results and control provenance. Execution used 79,582 transitions, 160.95 measured elapsed seconds and 164.28 total CPU-seconds, with zero experiment-model calls. Timing follows the saved controller fields; analysis and supervision are additional.

A separate passive audit replayed the already exposed 79-frame generation-2 example, without new world transitions. Removing its stale-cell reporting withdrawal added 46 reported frame-cell claims, only four correct. Accuracy fell from 97.8163% to 97.4401%, while all 79 actions stayed identical. This is a reporting-denominator effect on one historical trajectory, not a population estimate or evidence of better control.

4.7 Independent-search study: first completed block

The frozen broader-study design compares three independent full-native searches with three original-seed rewrite searches. Each has a maximum of 24 provider calls across all roles, 5,400 provider seconds and two hours. Full-native search retains ancestry, islands, inspirations, two-model bandit selection, local embedding novelty and meta/prompt evolution. The rewrite arm draws independent full programs from the original seed with alternating models and no auxiliary calls. This compares whole search strategies; common ceilings do not imply equal realized computation. After all six searches, 64 fresh shared cases select frozen nominees and 512 further cases assess winners. Neither fresh pool has been drawn.

The first full-native repetition (full-r1) is complete. It evaluated the seed and seven valid descendants on the same five public training cases: 40 episodes, 2,869 transitions, zero invalid executions. The highest training fitness belongs to generation 5: F=0.943888, S=0.90648, A=1.000000, 5/5 escapes, with 17.07% final reported-map coverage. The seed had F=0.536362 and 0/5 escapes. These are training results under adaptive search; they do not establish generalization or search-strategy superiority. Perfect reported-cell accuracy with sparse coverage again illustrates the reconstruction objective's limitation.

First independent full-native search and all-role request allocation

Figure 14. First independent full-native search checkpoint. The incumbent curve uses the all-role call count when each proposal was accepted for evaluation. The right panel counts every dispatched provider request, including one timeout. All four islands were sampled, one migration occurred, four samples used inspirations and four used recommendations. Both mutation models were used; local embeddings, four novelty-judge calls and an evolved prompt were observed. These events establish mechanism execution, not their causal benefit.

The run stopped normally at 24 calls: nine mutation, four novelty, eight summary, one global-insight, one recommendation and one prompt-mutation request. One mutation request timed out; its usage is unknown and remains outside the reported token lower bound. All calls required subscription billing and passed the raw context audit; 23 returned reported subscription usage. Local CPU embeddings made nine calls separately. Execution consumed 44.93 measured wall-minutes, 43.38 provider-minutes and 1.10 GiB peak combined RSS. A final generated generation-8 proposal was held before its required novelty check because the call cap had been reached. It is preserved as unevaluated and cannot enter selection. Native lineage, sources, representative saved replays, resource and route audits are in the compact checkpoint. Native search guidance is exported as model-produced guidance, not validated scientific findings. A source inspection shows that sparse exported maps coexist with persistent internal terrain and enemy beliefs used by the planners. The risk propagation uses hardcoded transition assumptions, not learned dynamics. Generation 5 also includes conditional waiting near the exit. On these saved training cases its 0.0012 fitness advantage over generation 2 follows exactly from five additional mean steps, because the original reward assigns a positive step contribution. This arithmetic does not establish whether that waiting branch activated. One rewrite block was subsequently interrupted after four valid descendants; four planned searches and fresh selection/assessment remain unexecuted.

4.8 Interrupted rewrite block and accountability checkpoint

The first independent rewrite block was stopped during request6 after the user challenged allowance consumption and the clarity of scientific contributions. Four returned rewrites each escaped5/5training cases; generation4 has the best trainingF=0.943533. The seed and four descendants completed25episodes and1,849 transitions. Request3 hit the local480-second timeout; request6 was interrupted by the operator. This is an interrupted attempt, not a completed budget-matched comparison. Its measured runtime was30.20wall-minutes; summed request time was 29.46minutes. Programs, outcomes and audit and the operator checkpoint are preserved. No fresh selection/assessment or calibration followed.

The accountability review records the larger management problem: expensive model settings were not calibrated, diminishing returns were already evident, and headline experiment counts excluded supervising and delegated assistant work. Local metadata for 8October17:00 through9October13:00UTC shows two supervisor sessions and42subagent sessions with observed usage; both supervisors and16subagents used Ultra. Recorded uncached input plus output was3,342,956tokens for supervision and 7,903,680for subagents. These are partial local token records, not an authoritative weekly subscription meter. They cannot be converted to a weekly-limit percentage. No claim about productive versus unproductive tokens follows from the aggregate.

5. Discussion and limitations

The seed accurately reconstructs the cells it reports, but rarely escapes. Its greedy planner can become blocked or pursue an unsuitable remembered target. The initial characterization establishes room for behavioral improvement. The subsequent search and 32-case paired check demonstrate improved task performance on development cases. The frozen-program assessment confirms a large gain over the original seed on fresh cases from the same distribution. Search, development, selection and assessment scores remain separate; only the five-case search panel supplies native search fitness.

The original accuracy term has specific limitations. An agent can obtain high accuracy by retaining easy or recently observed cells while covering little of the maze. Static terrain also contributes many targets. The 0.6/0.4 coefficients do not guarantee balanced selection pressure, and the task reward gives a positive contribution to elapsed steps. We disclose these properties while preserving the requested objective. High composite fitness must not be described as proof of understanding.

The two historical successful descendants are direct seed mutations with no inspirations. The new independent run uses accumulated ancestry and inspirations, but neither observation establishes their causal benefit. Persistent memory is used by their planners, but the contribution of mapping versus pathfinding versus risk heuristics is only partly isolated by the two specified interventions in section 4.6. Independent discovery reliability remains unmeasured. The known uniform enemy law is used as a heuristic; these results do not show learned transition dynamics.

The seed comparison is deliberately faithful to the original proposal, but its greedy/random planner is a weak control. The subsequent competent-control experiment does not establish a fitness advantage over the handcrafted pathfinder. The selection results also provide no clear basis for preferring one successful descendant's behavior over the other's, despite the required scalar ranking.

6. Reproduction

Use the existing environment created by bootstrap.sh. Run the focused correctness checks:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m unittest tests.test_proposal

Re-evaluate the initial program into a new directory:

OPENBLAS_NUM_THREADS=1 .venv/bin/python proposal/evaluate.py \
  --program_path proposal/initial.py \
  --results_dir results/proposal-reproduction --episodes 32 --replay-first

Regenerate the README figures from the saved experiment, without executing agents:

OPENBLAS_NUM_THREADS=1 .venv/bin/python proposal/figures.py

See proposal/README.md for preparation and explicit native launch commands. The launcher defaults to preparation; --run is required for model calls. No historical campaign or assessment is resumed by these commands.

Rebuild the Stage 1 figures from the committed compact evidence, without the private runtime database or any new episodes/model calls:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.search_report --render-export

The original bounded invocation was .venv/bin/python proposal/evolve_full.py --run. Its saved failure gate and deadline prevent further model work; this is not an instruction to restart it. A future evolutionary continuation requires a new explicit resource grant and reconciliation of the held pre-source slot. The run state records the exact boundary. The local native dashboard can be reopened without model calls:

.venv/bin/python scripts/v3_webui.py \
  --campaign results/proposal-full-native-01 --port 8001

Rebuild the Stage 3 paired analysis and Chromatic Fields figures from saved data, without running episodes or models:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.development_report

Reproduce the frozen generation-2 evaluation into a new directory:

OPENBLAS_NUM_THREADS=1 .venv/bin/python proposal/evaluate_full.py \
  --campaign-manifest artifacts/proposal/stage3-development/protocol.json \
  --program_path artifacts/proposal/full-native-01/programs/generation-002.py \
  --results_dir results/proposal-stage3-reproduction \
  --episodes 32 --seed-start 0 --record-first yes

These are reproduction commands, not automatic authorization for another stage.

Rebuild the Stage 4 analysis and figures from committed ordinal episode data:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.selection_report

The exact Stage 4 execution command was:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.panel \
  --protocol artifacts/proposal/stage4-selection/protocol.json \
  --output results/proposal-stage4-selection

That bounded invocation requires the locally preserved private pool and retains its original deadline; it is not a command to launch a fresh panel. The runner checkpoints each episode and refuses to silently retry an interrupted one. Public statistical reproduction needs neither the private pool nor new episodes.

Rebuild the Stage 5 assessment analysis and Chromatic Fields figures using only the committed public records:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.assessment_report

The exact completed Stage 5 invocation was:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.assess \
  --protocol artifacts/proposal/stage5-assessment/protocol.json \
  --output results/proposal-stage5-assessment

Its deadline is closed. Preserve the private pool and checkpoints; do not redraw or rerun the assessment. The current checkpoint records recovery of the completed execution and the ongoing broader experimental study.

Rebuild the competent-control analysis and figure without new episodes or models:

OPENBLAS_NUM_THREADS=1 .venv/bin/python -m proposal.control_report

References

  1. Sakana ShinkaDreamer proposal. Preserved original project description and example code.
  2. Lange, R.T., Imajuku, Y., and Cetin, E. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution. Final ICLR 2026 paper; official implementation.

About

Evolutionary program search for agents that learn world models and plan in partially observed dynamic mazes. ShinkaEvolve jointly evolves predictive representations, online learning rules, and planners, with controlled experiments on prediction, decision-making, and generalization.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages