EOS26

Description

All scripts to run the EOS26 simulation with 21cmFASTv4.2. EOS26 is run phase by phase i.e. ICs first, then PFs, then PHFs, and finally coevals.

How to run

All scripts accept an optional --test flag to run a small test box (HII_DIM=200).

Comparison workflow

• Generate the reference once with --test and no --compare. It is written to EOS26_test_HIIDIM200/.
• Run the production workflow with --compare to check each output against that reference.
• For a local workflow check, use both flags: --test --compare. Candidate files then go to EOS26_test_HIIDIM200_compare/, so the reference is never overwritten. Both test modes use the same input parameters and random seed (42).

End-to-end test

• sbatch sbatch_scripts/full_test_job.sh — runs all three pipeline steps sequentially in test mode.
• qsub pbs_scripts/full_test_job.sh — PBS equivalent of the end-to-end test.

Scaling Measurements

Peak memory (and time) of every phase as a function of HII_DIM and N_THREADS, for 21cmFAST v4.2 and v4.3, to find the largest HII_DIM that fits safely on a 3 TB megamem node. Full description: scaling/README.md.

• bash scaling/campaign_v4.2.sh / bash scaling/campaign_v4.3.sh submit one PBS job per (HII_DIM, N_THREADS). Re-running a campaign script resumes unfinished configurations from the beginning of their unfinished phase.
• python3 scaling/summarize.py prints the status/results table and writes scaling/results/<version>/summary_*.csv.
• The v4.3 environment is created with envs/create_venv_v4.3.sh (Mac or Gadi). Switch versions with source envs/use.sh v4.2|v4.3.
• The previous harness and its v4.2 results are in scaling_old/. They are superseded; see scaling_old/ASSESSMENT.md.

Production run (in order)

1. Initial conditions (ICs):
   • sbatch sbatch_scripts/ICs_job.sh [--test]

• qsub pbs_scripts/ICs_job.sh [--test] on PBS.
• Writes EOS26.toml (full template with embedded node redshifts and random seed) and EOS26_minimal.toml, then runs run_scripts/run_ICs.py.

2. Perturbed fields (PFs):
   • bash sbatch_scripts/submit_PF_jobs.sh [--test] — submits one job per PF (indices 0–91) (not used for production).
   • sbatch sbatch_scripts/N_PF_job.sh <z_idx> [N] [--test] — runs a batch of N PFs (default N=10) starting from redshift index z_idx.

• bash pbs_scripts/submit_PF_jobs.sh [--test] and qsub pbs_scripts/N_PF_job.sh <z_idx> [N] [--test] are the PBS equivalents.

3. Perturbed halo fields (PHFs):
   • sbatch sbatch_scripts/PHFs_job.sh [--test] — runs run_scripts/run_PHFs.py.

• qsub pbs_scripts/PHFs_job.sh [--test] on PBS.

4. Coevals:
   • sbatch sbatch_scripts/N_coeval_job.sh [N] [--test] — runs a batch of N coevals (default N=10) with run_scripts/run_N_coevals.py.

• qsub pbs_scripts/N_coeval_job.sh [N] [--test] on PBS.

Table

EOS26 simulation step	Computation time [hrs]		Memory [TB]		Storage [TB]		SUs
	Estimated	Actual	Estimated	Actual	Estimated	Actual	Estimated	Actual
Initial conditions	17+~3 write	30	2.63	2.41	1.42	1.57
One perturbed field	1.42	0.2	0.545	0.505	--Gb x 92 = 2.3Tb	2.4Tb
Perturbed halo fields	22 hrs		0.71	0.77	~330 G	33 G	720 EM	617 EM
Evolving astrophysics for one coeval	7		3.65		2.6Tb x 92 = 240 Tb 0.215 x 92 = 20Tb without XRS		672 EM x 92 = 62k

Scaling test results

Note (2026-09-30): the values below come from the superseded v4.2 measurements in scaling_old/ (the HII_DIM=500 coeval point covers only 5/92 coevals; see scaling_old/ASSESSMENT.md). They will be replaced by the results of the new scaling/ campaigns.

When updated with --update-readme, each extrapolated value is the mean ± 1σ from all available scaling points. Each extrapolated quantity shows both its current model (affine overhead + coefficient * HII_DIM^3 for peak RSS; free-exponent power law for time and storage) and a fixed-cubic a=3 fit. The regression uncertainty is combined in quadrature with a fixed 10% relative measurement-uncertainty floor. EOS-1: HII_DIM = 1500 (1.5 cMpc/cell, 2250 Mpc). EOS-2: HII_DIM = 1200 (1.667 cMpc/cell, 2000 Mpc). Storage for PFs and coevals is the total across all 92 node redshifts; coeval storage includes retained IonizedBox and excludes transient XraySourceBox. The scaling-test fits are trained only on HII_DIM<=500 and, for initial conditions, that range's fixed per-job overhead makes the fitted compute-time noticeably underestimate the production values now measured at both EOS target sizes (elapsed_seconds fit for HII_DIM=1500: 17h vs. 29.8h measured).

EOS26 simulation step	Computation time [hrs]		Memory		Storage
	N=100	N=500	N=100	N=500	N=100	N=500
Scaling tests (measured)
Initial conditions	0.0374	0.624	0.939 GB	97.2 GB	0.464 GB	58 GB
One perturbed field	0.0345	0.0447	0.334 GB	20.2 GB	0.016 GB	2 GB
Perturbed halo fields	0.0498	0.659	0.564 GB	51 GB	0.207 GB	25.4 GB
Evolving astrophysics for one coeval	0.00199	0.423	1.16 GB	97.2 GB	0.0601 GB x 92 = 5.53 GB	n/a (partial run: 5/92 coevals)
Extrapolated to EOS (HII_DIM = 1500, 1.5 cMpc/cell, 2250 Mpc)
Initial conditions	a=1.72: 3.04 ± 2.21	a=3: 17 ± 1.83	affine (overhead + a=3): 2.62 ± 0.262 TB	a=3: 2.63 ± 0.263 TB	a=3: 1.57 ± 0.149 TB	a=3: 1.57 ± 0.157 TB
One perturbed field	a=0.152: 0.05 ± 0.0074	a=3: 1.42 ± 0.832	affine (overhead + a=3): 0.54 ± 0.054 TB	a=3: 0.546 ± 0.0547 TB	a=3: 0.0539 ± 0.00515 TB	a=3: 0.054 ± 0.0054 TB
Perturbed halo fields	a=1.6: 2.84 ± 2.21	a=3: 18.1 ± 2.07	affine (overhead + a=3): 1.37 ± 0.137 TB	a=3: 1.38 ± 0.138 TB	a=2.99: 0.675 ± 0.0648 TB	a=3: 0.686 ± 0.0686 TB
Evolving astrophysics for one coeval	a=3.26: 11 ± 7.78	a=3: 11.1 ± 1.36	affine (overhead + a=3): 2.59 ± 0.267 TB	a=3: 2.65 ± 0.272 TB	a=3: 0.202 ± 0.0193 TB x 92 = 18.6 ± 1.77 TB	a=3: 0.203 ± 0.0203 TB x 92 = 18.6 ± 1.86 TB

EOS Fixed-Cubic Runtime Plan

Planning values use fixed a=3 central estimates at HII_DIM=1500, a 1.5 h IC-read allowance per dependent job, and a 13 h IC-write time (measured production elapsed time minus the small-scale a=3 cubic fit's compute-only prediction at the same HII_DIM; see estimate_ic_write_hours) to infer output-write throughput. Coeval output includes retained IonizedBox and excludes transient XraySourceBox. The table uses the 48 h maximum job walltime; coeval jobs are deliberately limited to 3 outputs for margin. Peak RSS is the per-phase process estimate; full-phase output totals all 92 PFs or coevals as applicable.

Phase	Work unit	Units/job	Compute/unit [h]	IC read/job [h]	Write/unit [h]	Estimated job walltime [h]	Jobs for full phase	Serial phase walltime [h]	Peak RSS [TB]	Output, full phase [TB]
Initial conditions	all ICs	1	16.98	0.00	12.85	29.83	1	29.8	2.63	1.42
Perturbed fields	one PF	24	1.42	1.50	0.44	46.19	4	184.8	0.55	4.52
Perturbed halo fields	all halo fields	1	18.08	1.50	5.63	25.21	1	25.2	1.38	0.62
Coevals	one coeval	3	11.10	1.50	1.66	39.80	31	1233.7	2.65	16.95