Skip to content

DOOP: add CPU/GPU benchmark matrix and exact oracle comparison - #93

Merged
StarGazerM merged 2 commits into
mainfrom
feat/doop-suite-runner
Sep 24, 2026
Merged

StarGazerM merged 2 commits into
mainfrom
feat/doop-suite-runner

Conversation

@StarGazerM

@StarGazerM StarGazerM commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

Run the twelve-dataset corpus against the actual canonical Python query, with separately selected CPU, GPU baseline and opt-in GPU bitmap execution.

  • Translate the instantiated logical program conservatively to Soufflé, preserving recursion, multi-head rules, negation, supported filters and execution-only SPLIT semantics.
  • Build/run GPU workers in fresh processes; require synchronized unlimited fixedpoints, checked lifecycle calls and normal process exit. Kill only owned child process groups on timeout.
  • Add checked native cardinality/TSV export APIs using authoritative canonical index columns, inverse permutations and bounded host buffers; handle two-level FULL/HEAD storage without a second full GPU reconstruction.
  • Record separate build/load/run/export timing scopes and complete failure artifacts. Defaults: one warmup, three measured repetitions. No silent dataset skipping.
  • Record all 74 relation counts and export all 37 derived tuple sets. Distinguish 39 prepared inputs from the 37 inputs actually consumed by the canonical program.
  • Require matching query/input identities, complete coverage and exact sorted integer tuple sets against a reference; counts alone are insufficient. Without a reference, correctness is explicitly not_compared.
  • Reject empty-root benchmarks. Preserve completed execution evidence when reference comparison itself fails.

Verification

  • Integrated Python suite: 1,110 passed, 5 skipped; Ruff checks/formatting passed. A subsequent native-artifact isolation fix passed all three worker-process regressions (including a new regression reproduced failing before the fix).
  • All 12 real pinned datasets prepared successfully; each selected exactly one main method. Complete input inventory was measured, not inferred from archive sizes.
  • All twelve CPU datasets completed normally, each with all 74 relation counts and all 37 IDB exports. The canonical query produced 2,742,413 VPT tuples for Xalan through 438,477,770 for Jython; H2O consumed 33,024,599 input rows. These are measured canonical-query counts, not assertions of equality with the upstream guide. Validation used zero warmups/one repetition amid other validation activity and is not a performance comparison.
  • Complete baseline and bitmap DOOP GPU libraries compiled with clang++-20/CUDA 12.9.
  • Bounded native CUDA smoke passed exact tuple export checks for 10 relations / 37 tuples, including EDB export, column permutations and both recursive bitmap source roles.
  • Additional native boundary smoke: baseline and bitmap each matched 77,874 expected tuples exactly, exercising multiword bitmaps, negative int32 values, overlapping projections and the 65,536-row export chunk; normal process exit with a 256 MiB pool cap.
  • Reproduced the reference-comparison exception accounting bug before the fix and verified completed evidence survives afterward; empty-root rejection was also exercised.
  • A real complete Xalan GPU attempt in each mode exceeded a 1 GiB RMM pool cap before fixedpoint. Both recorded failure, with abnormal teardown exit; neither is counted as a successful benchmark or tuple match.
  • That failure exposed an allocator log written into the checkout. Workers now run from their external output directory; the artifact-isolation regression fails before the fix and passes afterward.

Validation blocker / draft status

An unrelated vLLM workload occupies most of the RTX 6000 Ada's VRAM (44,742 MiB used / 49,140 MiB total when checked). It was not stopped. Full twelve-dataset GPU equivalence and performance are NOT established; leave this PR draft until that matrix can run with sufficient GPU headroom. No published speedup claim.

Reproduce

See docs/benchmarks.md for preparation, CPU reference, GPU baseline/bitmap comparison, selection, compiler prerequisites, timing boundaries and process-timeout semantics. Keep data/build/results outside the checkout and use a new output directory per suite run.

Stack

Depends on #92, which follows #91. Review and merge in that order. No research archive, generated binary, fact file or result dump is included.

@StarGazerM
StarGazerM changed the base branch from feat/doop-bitmap-plan to main September 24, 2026 02:44
@StarGazerM
StarGazerM marked this pull request as ready for review September 24, 2026 02:44
@StarGazerM
StarGazerM merged commit 114331c into main Sep 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant