diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 61e18d7..7f014ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -143,3 +143,16 @@ jobs: curl --fail --location --retry 3 https://raw.githubusercontent.com/hyperpolymath/ANCOMBC/6d094731ed85e9b0458053ad3510e9bdc238e152/R/ancombc_bias_correct.R -o /tmp/ancombc-variance-repair.R export R_HOME="$(R RHOME)" ANCOMBC_ESTIMATOR_SOURCE=/tmp/ancombc-variance-repair.R julia --project=test/oracle test/oracle/ancombc_oracle.jl + - name: Investigate the small-unbalanced calibration failure + if: matrix.reference == 'ANCOMBC' + run: | + julia --project=test/oracle benchmark/export_calibration_case.jl /tmp/calibration/small-unbalanced-inputs + Rscript benchmark/calibration_reference.R /tmp/calibration/small-unbalanced-inputs /tmp/ancombc-variance-repair.R + - name: Preserve calibration investigation evidence + if: always() && matrix.reference == 'ANCOMBC' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: small-unbalanced-calibration + path: /tmp/calibration + if-no-files-found: error + retention-days: 30 diff --git a/.gitignore b/.gitignore index 16b9253..f07ef2d 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ *.agdai _build/ /test/oracle/Manifest.toml +/benchmark/Manifest.toml diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc new file mode 100644 index 0000000..1c06f1a --- /dev/null +++ b/EXPLAINME.adoc @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) += What CompositionalDA can establish + +This package analyses raw integer counts in samples-by-features orientation. +Its current methods are unpaired two-group ALDEx2 and original ANCOM-BC. +It does not implement ANCOM-BC2, covariate adjustment or repeated measures. +Zero handling, the contrast direction, seeds and numerical policies belong in +the analysis record. Swapping groups changes the effect direction. + +== Evidence and its limits + +[cols="2,3,3",options="header"] +|=== +|Claim |Owning evidence |What remains outside it +|Exact algebraic identities |`proofs/PROOF-STATUS.md`, safe Agda modules and negative controls +|Floating-point refinement, sorter correctness, FDR and biological truth +|Numerical method agreement |`test/oracle/`, pinned R/Bioconductor releases and explicitly repaired reference functions +|Equivalence to pristine defective references, every input or later upstream releases +|Regression resistance |`test/`, seeded fuzz/property tests and planted mutants +|Exhaustive path/state coverage or the absence of unknown defects +|Real processed microbiome inputs |`docs/DATASETS.md`, archived source hashes and gut/soil fixtures +|Known ground truth, prospective clinical validity or general calibration +|Exploratory operating characteristics |`docs/CALIBRATION-PILOT-PROTOCOL.adoc`, `benchmark/calibration_pilot.jl` +|A calibration certificate; the small-unbalanced ANCOM-BC null pilot is adverse +|Performance |`benchmark/benchmarks.jl`, machine-specific raw measurements +|A speed advantage over R or an agreed latency/service-level guarantee +|=== + +The 2026-11-07 calibration pilot fixed six regimes and 32 independently seeded +datasets per regime before measurement. Of 384 method fits, 365 returned and +19 refused. Rates conditional on returned fits must be accompanied by refusal +counts. Under the small-unbalanced complete null, original ANCOM-BC made false +discoveries in 19/29 successful fits at 0.05 (Wilson 95% interval 0.473–0.801). +The unconditional observed count is 19/32; refusals are not successful negative +findings. The follow-up repaired-R experiment converged for all 32 datasets and +also made false discoveries in 19. Of 29 paired successful native fits, 28 had +maximum q-value differences below 8.5e-6, but replicate 27 differed by 0.5041 +and changed the discovery decision. Numerical conformance is therefore not +established across this expanded regime. A reviewed support restriction or +estimator correction, and fresh prespecified validation, are needed before a +calibration claim. The raw receipt is CI run 42411431729, artifact +`small-unbalanced-calibration`, at source `cbb8ac4d85b4259c8122ed83c2cce8ec6c34362a`. +Do not fix it by tuning the pilot seeds, removing adverse regimes or quietly +substituting another method. + +ALDEx2's pilot is much less adverse, but 32 datasets cannot certify 5% FDR. +All-null FDR is the probability of any rejection. With signal present, +family-wise error is the probability of at least one false rejection; mean +false discovery proportion estimates FDR. The reported Wilson intervals are +for event probabilities, not mean FDP, taxa treated as independent +replications, or arbitrary populations. + +At only two ALDEx2 Monte-Carlo instances, a separate 4,096-seed probe produced +58 infinite and three undefined effect estimates. Its p/q bounds still held. +The WebUI refuses non-finite effects; this is an estimator usability limitation +to address explicitly, not permission to coerce undefined effects to zero. + +== Trust and release decisions + +Julia, GMP/MPFR, BLAS, dependencies, the operating system, reference software, +data provenance and proof checkers remain in the trusted environment. +Agda algebra does not independently verify those implementations. No second +Rocq/Lean witness or full executable refinement is claimed. + +General registration is explicitly on hold pending the human maintainer's review +of the changed implementation. A signature authenticates a commit, not its +mathematical or scientific correctness. Version 0.3.0 in Project.toml is not a +stable-release grade. No CRG/TRG/RSR certificate is asserted. diff --git a/README.md b/README.md index bc115d1..8522287 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,15 @@ backend. > gate now checks all statistics and shared Monte-Carlo instances. ANCOM-BC's > precise model and limitations are described in `docs/ANCOMBC-SPEC.md`. +> **Launch review remains pending.** The predeclared calibration pilot found +> severe false-discovery inflation for original ANCOM-BC with four versus +> sixteen samples under a complete null: 19 of 29 successful fits made at least +> one discovery at q <= 0.05; three additional fits refused to return a result. +> This is a release blocker for a general calibration claim. See +> [`docs/LAUNCH-READINESS.adoc`](docs/LAUNCH-READINESS.adoc), the reproducible +> protocol and scripts in `benchmark/`, and [`EXPLAINME.adoc`](EXPLAINME.adoc). +> Passing numerical reference comparisons does not establish FDR control. + The v0.3 implementation, tests, proofs and integration work were developed with OpenAI Codex assistance. General registration requires a human maintainer to review and understand these contributions; automated checks and code review diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..4933997 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,25 @@ + + + +# Security and numerical integrity + +This is a development package; no released version currently carries a +security support commitment. Report exploitable defects through GitHub's +private vulnerability-reporting facility if enabled. Otherwise contact the +maintainer at the address in Project.toml before publishing exploit details. +Ordinary numerical defects can be reported as issues with a minimal synthetic +reproducer, package/tool versions and expected versus observed results. +Do not attach confidential or identifiable biological data. + +The native API is a local scientific library, not a sandbox for untrusted jobs. +Callers must bound matrix dimensions, count bit widths, Monte-Carlo work, +concurrency and memory at their service boundary. Input validation and explicit +nonconvergence errors do not create a hard CPU or memory limit. R reference +execution loads installed R packages and must run in a separately controlled +environment when inputs or dependencies are untrusted. + +Aqua checks Julia package quality. It is not a vulnerability-database scanner. +The launch ledger must separately record dependency advisories, secret scanning, +workflow permissions/action pins, release provenance and unresolved findings. +No successful comprehensive security audit is claimed here. Statistics that +fail calibration are safety/integrity concerns even without an attacker. diff --git a/benchmark/Project.toml b/benchmark/Project.toml new file mode 100644 index 0000000..edd428a --- /dev/null +++ b/benchmark/Project.toml @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +[deps] +BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf" +CompositionalDA = "bb1787b7-c25e-80cd-8559-27e3f8d35816" +Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f" + +[compat] +BenchmarkTools = "1" +Distributions = "0.25" +julia = "1.10" diff --git a/benchmark/README.md b/benchmark/README.md new file mode 100644 index 0000000..a088e91 --- /dev/null +++ b/benchmark/README.md @@ -0,0 +1,52 @@ + + + +# Exploratory launch measurements + +From the repository root, prepare the separate environment: + +```sh +julia --project=benchmark -e 'using Pkg; Pkg.develop(path="."); Pkg.instantiate()' +julia --project=benchmark benchmark/calibration_pilot.jl /path/to/pilot-output +julia --project=benchmark benchmark/benchmarks.jl /path/to/benchmark-output +``` + +Read `docs/CALIBRATION-PILOT-PROTOCOL.adoc` before interpreting the output. +Seeds, scenarios and comparisons were fixed before results. Refusals are +separate from successful fits. Neither script certifies calibration, speed, + + + + +stability or package readiness. Keep generated measurements outside source. +Each output directory includes the active environment's `Project.toml` and +resolved `Manifest.toml`, including transitive dependencies such as StatsFuns. +Retain both with the results; instantiate the active project before running. +Timing benchmarks refuse a known dirty source tree (including untracked files). +When Git provenance is unavailable, scripts warn and record `source_head` and +`source_tree` as `unknown`; those runs need a separate source archive to reproduce. + + +Benchmark timings exclude compilation/warmup and retain trial counts; ten +trials provide an exploratory baseline, not precise tail-latency estimates. + +The optional CI investigation exports the small-unbalanced complete-null +inputs and native results, then runs pinned ANCOMBC 2.12.0 with the explicitly +identified variance-estimator repair. It records R convergence rather than +treating an iteration-cap result as a valid fit. A successful investigation +job means the experiment completed, not that the adverse calibration passed. + + + + + + + + +The workflow requests 30 days of artifact retention, but repository policy +may shorten it. Check the artifact expiry and archive it with its SHA-256 +before using it in a release decision. + + + + diff --git a/benchmark/benchmarks.jl b/benchmark/benchmarks.jl new file mode 100644 index 0000000..50ba0a2 --- /dev/null +++ b/benchmark/benchmarks.jl @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +using BenchmarkTools, CompositionalDA, Statistics, TOML, Dates +include("calibration_pilot.jl") +destination=only(ARGS) +mkpath(destination) +provenance=source_provenance(pkgdir(CompositionalDA);require_clean=true) +save_run_environment(destination) +rows=NamedTuple[] +for (n,d) in ((20,40),(20,400),(100,400)) + regime=(name="benchmark",n1=n÷2,n2=n÷2,depth1=10000,depth2=10000,sd=.35,sparse=false,signals=4) + x,f,g,_=simulate(regime,20261107+n+d;d) + for method in (:closure,:clr,:bh,:aldex2,:ancombc) + p=collect(range(.001,.999;length=d)) + call = method==:closure ? (() -> closure(x)) : + method==:clr ? (() -> clr(Pseudocount(1),x)) : + method==:bh ? (() -> bh_adjust(p)) : + method==:aldex2 ? (() -> aldex2(x,f,g;mc_samples=128,seed=123)) : + (() -> ancombc(x,f,g;structural_zeros=false)) + try + call() # compilation/warmup is excluded and labelled as such + trial=@benchmark $call() samples=10 evals=1 seconds=15 + push!(rows,(method=String(method),samples=n,features=d,status="measured", + trials=length(trial.times),median_ns=median(trial.times),p95_ns=quantile(trial.times,.95), + bytes=trial.memory,allocations=trial.allocs)) + catch error + error isa CompositionError || rethrow() + push!(rows,(method=String(method),samples=n,features=d,status="refused", + trials=0,median_ns=NaN,p95_ns=NaN,bytes=0,allocations=0)) + println("REFUSED ",method," ",n,"x",d,": ",sprint(showerror,error)) + end + end +end +open(joinpath(destination,"benchmarks.csv"),"w") do io + println(io,join(keys(first(rows)),',')) + foreach(row->println(io,join(values(row),',')),rows) +end +metadata=Dict("julia"=>string(VERSION),"threads"=>Threads.nthreads(),"cpu"=>Sys.CPU_NAME, + "machine"=>Sys.MACHINE,"word_size"=>Sys.WORD_SIZE,"date_utc"=>string(now(UTC)), + "benchmarktools"=>string(pkgversion(BenchmarkTools)),"warmup_excluded"=>true, + "claim"=>"exploratory warm-process baseline; no reference speedup or acceptance budget") +merge!(metadata,provenance) +open(io->TOML.print(io,metadata),joinpath(destination,"benchmarks-metadata.toml"),"w") diff --git a/benchmark/calibration_pilot.jl b/benchmark/calibration_pilot.jl new file mode 100644 index 0000000..f21677c --- /dev/null +++ b/benchmark/calibration_pilot.jl @@ -0,0 +1,119 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Run with the package project; results are written only to the supplied directory. +using CompositionalDA, Random, Statistics, Dates, Printf, SHA, TOML +using CompositionalDA: StableRNG +using Distributions: Multinomial +include("run_environment.jl") + +const REGIMES = [ + (name="balanced_null",n1=10,n2=10,depth1=10000,depth2=10000,sd=0.35,sparse=false,signals=0), + (name="depth_imbalance_null",n1=10,n2=10,depth1=1000,depth2=10000,sd=0.35,sparse=false,signals=0), + (name="small_unbalanced_null",n1=4,n2=16,depth1=1000,depth2=1000,sd=0.35,sparse=false,signals=0), + (name="sparse_null",n1=6,n2=6,depth1=200,depth2=200,sd=0.7,sparse=true,signals=0), + (name="sparse_signal",n1=10,n2=10,depth1=10000,depth2=10000,sd=0.35,sparse=false,signals=4), + (name="dense_balanced_signal",n1=10,n2=10,depth1=10000,depth2=10000,sd=0.35,sparse=false,signals=20), +] + +function simulate(regime,seed;d=40) + rng=StableRNG(seed) + baseline=regime.sparse ? exp.(range(-4,0;length=d)) : collect(range(1000,5000;length=d)) + logeffect=zeros(d) + half=regime.signals÷2 + logeffect[1:half].=log(4) + logeffect[half+1:2half].=-log(4) + x=Matrix{Int}(undef,regime.n1+regime.n2,d) + for i in axes(x,1) + condition=i>regime.n1 + weights=baseline .* exp.(regime.sd .* randn(rng,d) .+ condition .* logeffect) + x[i,:]=rand(rng,Multinomial(condition ? regime.depth2 : regime.depth1,weights/sum(weights))) + end + return x,string.(1:d),vcat(fill(0,regime.n1),fill(1,regime.n2)),logeffect +end + +function wilson(events,total) + total==0 && return (NaN,NaN) + z=1.959963984540054; p=events/total; scale=1+z^2/total + centre=(p+z^2/(2total))/scale + radius=z*sqrt(p*(1-p)/total+z^2/(4total^2))/scale + return (max(0,centre-radius),min(1,centre+radius)) +end + +function run_pilot(destination) + mkpath(destination) + provenance=source_provenance(pkgdir(CompositionalDA)) + save_run_environment(destination) + protocol=joinpath(@__DIR__,"..","docs","CALIBRATION-PILOT-PROTOCOL.adoc") + metadata=Dict("protocol_sha256"=>bytes2hex(sha256(read(protocol))), + "julia"=>string(VERSION),"package_version"=>string(pkgversion(CompositionalDA)), + "started_utc"=>string(now(UTC)),"replicates_per_regime"=>32, + "mc_samples"=>128,"seed_formula"=>"20261107 + 10000*regime_index + replicate", + "status"=>"exploratory pilot; not a calibration certificate") + merge!(metadata,provenance) + records=NamedTuple[] + failures=NamedTuple[] + for (regime_index,regime) in enumerate(REGIMES), replicate in 1:32 + seed=20261107+10000regime_index+replicate + x,features,groups,truth=simulate(regime,seed) + for method in (:aldex2,:ancombc) + start=time_ns() + fitted=try + method==:aldex2 ? aldex2(x,features,groups;mc_samples=128,seed=seed+1_000_000) : + ancombc(x,features,groups;prevalence=0.1,structural_zeros=false,conserve=true) + catch error + if error isa CompositionError + push!(failures,(regime=regime.name,replicate,method=string(method),seed, + reason=sprint(showerror,error),seconds=(time_ns()-start)/1e9)) + continue + end + rethrow() + end + elapsed=(time_ns()-start)/1e9 + table=fitted.table + @assert all(0 .<= table.pvalue .<= table.qvalue .<= 1) + ids=parse.(Int,table.feature) + for alpha in (0.05,0.10) + discovered=table.qvalue .<= alpha + false_calls=count(discovered .& iszero.(truth[ids])) + true_calls=count(discovered .& .!iszero.(truth[ids])) + push!(records,(regime=regime.name,replicate,method=string(method),seed,alpha, + tested=length(ids),excluded=40-length(ids),false_calls,true_calls, + fdp=false_calls/max(false_calls+true_calls,1), + power=regime.signals==0 ? NaN : true_calls/regime.signals,seconds=elapsed)) + end + end + replicate==32 && println("Completed ",regime.name) + flush(stdout) + end + open(joinpath(destination,"calibration-pilot.csv"),"w") do io + println(io,join(string.(keys(first(records))),',')) + for record in records + println(io,join(values(record),',')) + end + end + metadata["finished_utc"]=string(now(UTC)) + metadata["successful_fits"]=length(records)÷2 + metadata["refused_fits"]=length(failures) + metadata["refusals"]=[Dict(string(k)=>v for (k,v) in pairs(f)) for f in failures] + open(io->TOML.print(io,metadata),joinpath(destination,"calibration-pilot-metadata.toml"),"w") + open(joinpath(destination,"calibration-pilot-summary.md"),"w") do io + println(io,"# Exploratory calibration pilot\n\n32 independent datasets per regime. Metrics condition on successful fits; refusals are separate. No calibration certification.\n") + println(io,"| Regime | Method | q threshold | Fits/32 | Mean FDP | Mean power | Any false discovery: fraction [95% Wilson interval] |") + println(io,"| --- | --- | --- | --- | --- | --- | --- |") + for regime in REGIMES,method in ("aldex2","ancombc"),alpha in (0.05,0.10) + rows=filter(r->r.regime==regime.name && r.method==method && r.alpha==alpha,records) + isempty(rows) && (println(io,"| $(regime.name) | $method | $alpha | 0/32 | — | — | — |");continue) + events=count(r->r.false_calls>0,rows) + lo,hi=wilson(events,length(rows)) + power=regime.signals==0 ? "N/A" : @sprintf("%.3f",mean(r.power for r in rows)) + @printf(io,"| %s | %s | %.2f | %d/32 | %.3f | %s | %.3f [%.3f, %.3f] |\n", + regime.name,method,alpha,length(rows),mean(r.fdp for r in rows),power,events/length(rows),lo,hi) + end + println(io,"\nFor complete-null regimes, any false discovery equals the false-discovery-rate event. For signal regimes that event is FWER, not mean FDP. The interval covers the event probability among successful fits, not arbitrary populations or the mean FDP. See the fixed protocol and refusal records.") + end + println("Successful fits: ",metadata["successful_fits"],"; refusals: ",length(failures)) +end + +if abspath(PROGRAM_FILE)==(@__FILE__) + run_pilot(isempty(ARGS) ? error("Supply an output directory") : only(ARGS)) +end diff --git a/benchmark/calibration_reference.R b/benchmark/calibration_reference.R new file mode 100644 index 0000000..ae2ba82 --- /dev/null +++ b/benchmark/calibration_reference.R @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +args <- commandArgs(trailingOnly=TRUE) +input <- args[[1]] +loadNamespace("ANCOMBC") +stopifnot(as.character(packageVersion("ANCOMBC")) == "2.12.0") +replacement <- new.env(parent=asNamespace("ANCOMBC")) +sys.source(args[[2]], envir=replacement) +assignInNamespace(".bias_em",replacement$.bias_em, ns="ANCOMBC") +results <- list() +for (replicate in seq_len(32)) { + reads <- as.matrix(read.delim(file.path(input,paste0("replicate-",replicate,".tsv")),row.names=1,check.names=FALSE)) + meta <- data.frame(condition=factor(c(rep("reference",4),rep("contrast",16)),levels=c("reference","contrast")), + unused=seq_len(20),row.names=colnames(reads)) + diagnostics <- list() + trace(".bias_em",where=asNamespace("ANCOMBC"),print=FALSE,exit=quote({ + .GlobalEnv$diagnostics[[length(.GlobalEnv$diagnostics)+1L]] <- c(iterations=iterNum,residual=epsilon) + })) + fit <- tryCatch(ANCOMBC::ancombc(data=reads,aggregate_data=reads,meta_data=meta, + tax_level=NULL,formula="condition",p_adj_method="BH",pseudo=1,prv_cut=.1,lib_cut=0, + group="condition",struc_zero=FALSE,neg_lb=FALSE,tol=1e-7,max_iter=20000, + conserve=TRUE,global=FALSE,n_cl=1,verbose=FALSE),error=function(e)e) + untrace(".bias_em",where=asNamespace("ANCOMBC")) + if(inherits(fit,"error")) { + results[[replicate]] <- data.frame(replicate,status=conditionMessage(fit),false_calls=NA,iterations=NA,residual=NA,native_false_calls=NA,max_q_gap=NA) + } else { + diag <- do.call(rbind,diagnostics) + converged <- nrow(diag)>0 && all(diag[,"iterations"]<20000) && all(diag[,"residual"]<=1e-7) + q <- fit$res$q_val$conditioncontrast + native_file <- file.path(input,paste0("native-",replicate,".csv")) + native_calls <- max_q_gap <- NA_real_ + if (file.exists(native_file)) { + native <- read.csv(native_file,colClasses=c(feature="character")) + ids <- if("taxon" %in% names(fit$res$q_val)) as.character(fit$res$q_val$taxon) else rownames(fit$res$q_val) + stopifnot(setequal(ids,native$feature),all(is.finite(q))) + native_calls <- sum(native$qvalue<=.05) + max_q_gap <- max(abs(native$qvalue-q[match(native$feature,ids)])) + } + results[[replicate]] <- data.frame(replicate,status=if(converged) "converged" else "unconverged", + false_calls=sum(q<=.05,na.rm=TRUE),iterations=max(diag[,"iterations"]),residual=max(diag[,"residual"]),native_false_calls=native_calls,max_q_gap) + } + write.csv(do.call(rbind,results),file.path(dirname(input),"small-unbalanced-repaired-R.csv"),row.names=FALSE) + cat("Finished replicate",replicate,"\n"); flush.console() +} diff --git a/benchmark/export_calibration_case.jl b/benchmark/export_calibration_case.jl new file mode 100644 index 0000000..3bee2a2 --- /dev/null +++ b/benchmark/export_calibration_case.jl @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +include("calibration_pilot.jl") +destination=only(ARGS) +mkpath(destination) +save_run_environment(destination) +for replicate in 1:32 + x,features,groups,truth=simulate(REGIMES[3],20261107+30000+replicate) + open(joinpath(destination,"replicate-$replicate.tsv"),"w") do io + println(io,join(vcat("feature",["s$i" for i in axes(x,1)]),'\t')) + for j in axes(x,2) + println(io,join(vcat(features[j],string.(x[:,j])),'\t')) + end + end + try + fitted=ancombc(x,features,groups;prevalence=.1,structural_zeros=false,conserve=true) + open(joinpath(destination,"native-$replicate.csv"),"w") do io + println(io,"feature,pvalue,qvalue,se,effect") + for row in eachrow(fitted.table) + println(io,join((row.feature,row.pvalue,row.qvalue,row.se,row.effect),',')) + end + end + catch error + error isa CompositionError || rethrow() + write(joinpath(destination,"native-refused-$replicate.txt"),sprint(showerror,error)) + end +end diff --git a/benchmark/run_environment.jl b/benchmark/run_environment.jl new file mode 100644 index 0000000..15ef3cf --- /dev/null +++ b/benchmark/run_environment.jl @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +using Pkg + +function source_provenance(source; require_clean=false) + head, state = try + head = strip(read(`git -C $source rev-parse HEAD`, String)) + status = read(`git -C $source status --porcelain --untracked-files=all`, String) + (head, isempty(status) ? "clean" : "dirty") + catch error + @warn "Source revision could not be captured; Git provenance is unavailable" exception=(error, catch_backtrace()) + ("unknown", "unknown") + end + require_clean && state == "dirty" && + error("Refusing to benchmark a dirty source tree; commit source changes and keep outputs outside the repository") + return Dict("source_head" => head, "source_tree" => state) +end + +function save_run_environment(destination) + # EnvCache selects the manifest used by Pkg, including version-specific names. + environment = Pkg.Types.EnvCache() + isfile(environment.manifest_file) || + error("A resolved manifest is required; instantiate the active project before running") + cp(environment.project_file, joinpath(destination, "Project.toml"); force=true) + cp(environment.manifest_file, joinpath(destination, "Manifest.toml"); force=true) +end diff --git a/benchmark/test_run_environment.jl b/benchmark/test_run_environment.jl new file mode 100644 index 0000000..4b6aae3 --- /dev/null +++ b/benchmark/test_run_environment.jl @@ -0,0 +1,46 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +using Test, TOML +include("run_environment.jl") + +@testset "Run provenance" begin + mktempdir() do source + run(`git -C $source init -q`) + write(joinpath(source,"source.jl"),"original\n") + run(`git -C $source add source.jl`) + run(`git -C $source -c user.name=Test -c user.email=test@example.invalid commit -qm baseline`) + head=strip(read(`git -C $source rev-parse HEAD`,String)) + @test source_provenance(source;require_clean=true)==Dict("source_head"=>head,"source_tree"=>"clean") + write(joinpath(source,"source.jl"),"modified\n") + @test source_provenance(source)["source_head"]==head + @test_throws ErrorException source_provenance(source;require_clean=true) + run(`git -C $source checkout -- source.jl`) + write(joinpath(source,"untracked.jl"),"new source\n") + @test_throws ErrorException source_provenance(source;require_clean=true) + withenv("PATH"=>"") do + provenance=@test_logs (:warn,r"Source revision could not be captured") source_provenance(source;require_clean=true) + @test provenance==Dict("source_head"=>"unknown","source_tree"=>"unknown") + end + end + mktempdir() do source + provenance=@test_logs (:warn,r"Source revision could not be captured") source_provenance(source) + @test provenance["source_head"]=="unknown" + end +end + +@testset "Resolved environment snapshot" begin + mktempdir() do destination + save_run_environment(destination) + environment=Pkg.Types.EnvCache() + @test read(joinpath(destination,"Project.toml"))==read(environment.project_file) + @test read(joinpath(destination,"Manifest.toml"))==read(environment.manifest_file) + manifest=TOML.parsefile(joinpath(destination,"Manifest.toml")) + @test haskey(only(manifest["deps"]["StatsFuns"]),"version") + end + mktempdir() do project + Pkg.activate(project) do + write(joinpath(project,"Project.toml"),"[deps]\n") + @test_throws ErrorException save_run_environment(project) + end + end +end diff --git a/docs/CALIBRATION-PILOT-PROTOCOL.adoc b/docs/CALIBRATION-PILOT-PROTOCOL.adoc new file mode 100644 index 0000000..2c54380 --- /dev/null +++ b/docs/CALIBRATION-PILOT-PROTOCOL.adoc @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) += Calibration pilot protocol + +This protocol is fixed before examining pilot results. It is exploratory +evidence for planning a larger independent study, not a calibration certificate +or a release acceptance test. No tolerance is tuned to observed results. + +== Estimands and regimes + +Generate independent samples with 40 absolute-abundance components, lognormal +biological variation and multinomial sampling at a specified library depth. +Use baseline abundances spanning 1,000–5,000, with log-scale SD 0.35 unless +specified. Conditions are labelled 0 and 1. Signal scenarios apply equal +numbers of fourfold increases and fourfold decreases, so the sum of log fold +changes is zero. In that restricted construction, population CLR differences +and absolute log-abundance differences have the same zero/nonzero support. +This alignment is intentional; it does not make these estimands equivalent +on arbitrary data. + +Run 32 independent datasets per regime, seeds determined before execution: + +* Balanced null: 10/10 samples, depth 10,000, no group effect. +* Depth imbalance null: 10/10, depths 1,000/10,000, no abundance effect. +* Small unbalanced null: 4/16, depth 1,000, no abundance effect. +* Sparse null: 6/6, depth 200, baseline abundances geometrically spanning + exp(-4)–1, biological log SD 0.7, no abundance effect. +* Sparse signal: 10/10, depth 10,000, two features up fourfold and two down + fourfold; 36 null features. +* Dense balanced signal: 10/10, depth 10,000, ten features up and ten down + fourfold; 20 null features. Treat this as sensitivity to the bias estimator's + predominantly-null identification premise. + +ALDEx2 uses 128 Monte Carlo instances and a separately recorded seed. ANCOM-BC +uses its documented conservative setting, prevalence 0.1, structural-zero +decisions disabled, tolerance 1e-7 and maximum 1,000 iterations. Refusals are +recorded separately and are never interpreted as successful zero discoveries. +Analyse reported q-values at 0.05 and 0.10. Record true/false discoveries, +false-discovery proportion, power, tested/excluded features and runtime. + +== Uncertainty and interpretation + +Report the fit-success fraction and metrics conditional on successful fits. +For each null regime, report the probability of at least one false discovery +(equal to FDR under the complete null) with a 95% Wilson interval over +independent datasets, not over correlated features. For signal scenarios +report mean false-discovery proportion and power, with transparent sample +counts; avoid a narrow-confidence claim from this pilot. + +Thirty-two replicates cannot certify a 5% false-positive bound: even zero +events leaves a substantial upper confidence limit. The study does not cover +all distributions, batch effects, arbitrary zero mechanisms, repeated +measures, ecological identification, external deployment or user acceptance. +The six scenarios are not six unrelated external CRG validation targets. +Reference-program agreement is evaluated separately on the existing real +gut/soil fixtures and mock data; it is not substituted for calibration here. + +== Provenance and reuse + +The estate's proven-tests-and-benches framework was inspected first. It +provides evidence/coverage abstractions, not a ready-made Julia compositional +calibration experiment for this estimand. This domain-specific pilot therefore +records its own assumptions, seeds, results and limitations. Its synthetic +data contain no human records. Preserve the protocol hash alongside results. diff --git a/docs/LAUNCH-READINESS.adoc b/docs/LAUNCH-READINESS.adoc new file mode 100644 index 0000000..0a3a9e9 --- /dev/null +++ b/docs/LAUNCH-READINESS.adoc @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) += Launch readiness: evidence before labels + +Assessment date: 2026-11-07. Candidate numerical source: +`911989080145b9ab43ce06a8c1c8dfe6712ca2e3`. +This is the author's assessment, not independent scientific review. +Registration remains on hold. No alpha/beta/stable or RSR certification is granted. + +== Source authority + +Estate standards were inspected at `fe57d352f0b1a50e5cb07b4aac2581caf355388e`, +gitbot-fleet at `46d26a4b084c40847d9a2f2ef98305502b7320bc`, and +proven-tests-and-benches at `e82c54ea6ef0e004f759f3533ebe8373e517f9e3`. +Refresh these before release. Joshua's CONTRIBUTING governs the WebUI contribution. +Current estate metadata canon uses one chora DEED, not retired A2ML/SCM files. +Conflicting old examples must not be copied as if authoritative. + +CRG assesses component readiness; TRG imposes additional language-toolchain +requirements. This scientific library is not a compiler/runtime merely because +it is written in Julia. SemVer, General eligibility and readiness grades are +separate. Six simulation regimes do not meet CRG's diverse external-target bar. +The taxonomy is 18 test categories by 14 aspects; the companion evidence +framework still documents a 17-by-14 schema. Preserve that migration gap. + +== Gates and work allocation + +[cols="2,3,3",options="header"] +|Gate or owner |Required evidence |Current disposition +|Finishbot (finishingbot) |README/EXPLAINME, install example, licences, About/topics, links, changelog and release description all agree with source +|About/topics corrected; explanation/security/launch docs proposed. Wiki, changelog, native estate metadata and final release packet remain pending. +|Seambot |Counts/orientation/groups, zero policy, Julia/R, package/WebUI, JSON/CSV, registry/install seams each have owners and positive/negative contract tests +|Substantial adapter and export tests exist. Current fleet parser list has no Julia parser; do not report automated Julia seam coverage without an implemented bridge. +|Echidnabot |Claim-to-theorem map, checker versions, safe/axiom audit, negative controls, counterexamples and executable refinement limits +|Safe Agda and negative controls pass. Independent proof witnesses and floating-point refinement remain pending; backend inventory is not a successful proof run. +|Rhodibot / Hypatia |Canonical metadata checks, repository policy and machine-readable findings with source hashes +|Estate conformance not established; retired metadata examples are an unresolved shared-guidance problem. +|Cipherbot |Actual dependency/secret/security scans, pinned CI provenance and triaged findings +|No complete scanner receipt yet; Aqua is quality evidence only. +|Glambot / Accessibilitybot |Representative branding, correct alt text, legibility at small sizes, text alternatives and no invented assurance marks +|Social concept approved by user; production export dimensions/size and avatar/hero review remain pending. +|Sustainabot |Measured runtime, allocations, scaling, work budgets and reproducible hardware/runtime context +|Benchmark harness proposed; no environmental-impact claim or performance advantage established. +|RobotRepoAutomaton |Execute authorised repairs with regression evidence and preserve provenance +|May implement fixes; cannot approve its own scientific or human-review gate. +|Panicbot |Escalate severe integrity/security failures without suppressing evidence +|Adverse small-unbalanced ANCOM-BC calibration occurs in native and repaired R results. Expanded comparison also finds one large q-value disagreement. Both remain launch blockers for the corresponding claims. +|Human maintainer and independent scientist |Review actual diff by hand, assess applicability, data rights, uncertainty and accepted claims +|Pending; automated or neural reviews do not count as those decisions. +|=== + +These are proposed objectives for the real fleet roles, not claims that a bot +ran or supports every language/backend. The fleet's finishing documentation +still mentions retired six-file SCM metadata; it needs reconciliation with +current canon before that objective can be marked complete. + +== Stage boundaries + +* Research/pre-alpha: explicit scope/refusals, reproducible baseline, no unsupported + calibration/clinical/stable claim. Adverse findings stay visible. +* Alpha candidate: applicable CRG minimum and canonical repository requirements, + tested installation and user workflow, signed provenance, known-bug treatment, + and human approval of the exact bounded claim. Test count alone is insufficient. +* Beta candidate: independent reproduction and scientific review; adequately + powered, prespecified calibration; relevant external datasets/targets; mutation, + fuzz and seam campaigns; measured performance/support budgets and regression policy. +* Stable candidate: satisfy the applicable estate V1 gate, including all claimed + proof/test/benchmark/security obligations and artifact/claim parity. A green CI + icon or version number cannot substitute for these requirements. + +General submission additionally needs correct name/UUID/compat/licence, clean +installation without an unregistered dependency, declared AI assistance and the +human review General requires. Registration review, the waiting period, tagging +and release publication remain distinct events. Other registries need their own +adapters for namespace, licence, provenance, binaries, signing and review rules. + +== Risk and uncertainty + +Prioritise severity, exposure, detectability and confidence; retain the raw +evidence rather than treating an ordinal risk score as a probability. Calibration +failure outranks cosmetic completeness. Define a Six Sigma defect and opportunity +before collecting DPMO; dependent test assertions are not independent customer +opportunities. No Six Sigma capability is inferred from these tests. + +Budget independent datasets, not just taxon-level observations. Record model +misspecification, selection/refusal bias, Monte-Carlo uncertainty, software drift +and human applicability review separately. Each release claim needs an owner, +acceptance criterion fixed before measurement, receipt, expiry/retest trigger, +remaining uncertainty and a rollback/withdrawal decision. + +The comprehensive independent-review prompt and 18-by-14 ledger are maintained +in the review packet. They must remain capable of rejecting this assessment.