[yunus-dev-codecrafter]

closes #211
Measure all four bundled template generation paths through the trusted template pipeline with a bounded, offline harness that performs no dependency install, Git initialization, or network access. It reports deterministic environment/artifact metadata plus wall-clock statistics (mean, median, p95, min, max, standard deviation, coefficient of variation) so repeated runs on the same platform can be compared for regressions. Timings are not asserted in CI; the suite verifies coverage of every path, offline generation without node_modules, and structural reproducibility.

Summary

Adds a bounded, local performance benchmark harness for the project generator. The harness measures all four bundled template generation paths (basic-app, full-stack, smart-contract, api-service) through the same trusted template pipeline used by stellarforge new, and reports wall-clock statistics plus deterministic artifact measurements so repeated runs on the same platform can be compared for regressions.

The harness performs no dependency installation, no Git initialization, and no network access, so its timings reflect local generator cost rather than registry or install latency. It is deterministic in structure (fixed iteration/warmup counts, canonical template ordering, stable artifact counts) while explicitly avoiding absolute-timing assertions so it stays reliable in CI.

Related Issue

Closes #211

Scope

Completed:

• tests/helpers/generator-benchmark.ts — reusable harness core. Runs warmup plus bounded, timed iterations of generateProjectFromTemplate (resolved via resolveBundledTemplateRoot) into isolated temp directories, computes mean/median/p95/min/max/standard deviation/coefficient of variation, records generated file count and byte totals, and flags any node_modules output. Exposes deterministic fingerprint, JSON, and human-readable text serializers plus a bounded iteration resolver.
• tests/generator-benchmark.test.ts — Vitest suite (runs under npm test and standalone): measures every bundled path offline; verifies each template's documented artifacts are generated with the substituted project name and no node_modules; asserts structural reproducibility across two runs; and verifies iteration overrides are defaulted and clamped (bounded).
• tests/helpers/index.ts — re-exports the new benchmark helper.
• docs/contributing/testing.md — new "Performance Benchmarks" section documenting purpose, exclusions, how to run, the iteration override, and regression comparison.

Intentionally left out:

• No new npm script and no package.json/.changeset changes (protected surfaces not in scope; test-only work does not require a changeset).
• No modifications to protected generator source files (template-generation.ts, template-registry.ts, template-sources.ts, validation.ts); they are only imported.
• No absolute timing thresholds enforced in CI, by design, to avoid flaky failures on shared runners.

Testing

Harness runs offline against the bundled templates and is covered by the normal Vitest suite.

Commands run locally:

• npm run typecheck — passed
• npm run lint — passed
• npm run build — passed
• node node_modules/vitest/vitest.mjs run tests/generator-benchmark.test.ts — 4/4 passed

Standalone harness output (verbose reporter, STELLARFORGE_BENCHMARK_ITERATIONS=5):

Generator performance benchmark (offline template generation)
node=v24.21.0 platform=win32 arch=x64
iterations=5 warmupIterations=2

basic-app         mean=20.339ms  median=17.171ms  p95=33.223ms  min=16.953ms  max=33.223ms  stddev=6.443ms  cv=0.317  files=6  bytes=2783
full-stack        mean=27.309ms  median=27.074ms  p95=28.923ms  min=26.521ms  max=28.923ms  stddev=0.842ms  cv=0.031  files=10 bytes=5075
smart-contract    mean=18.546ms  median=17.004ms  p95=25.888ms  min=14.929ms  max=25.888ms  stddev=3.886ms  cv=0.210  files=6  bytes=2617
api-service       mean=30.025ms  median=25.931ms  p95=47.111ms  min=23.954ms  max=47.111ms  stddev=8.731ms  cv=0.291  files=9  bytes=8001

Notes:

• Generated file counts/bytes are stable across runs; median is the recommended comparable metric. Coefficients of variation reflect local Windows disk/AV variance and are reported, not asserted.

• npm test locally shows two unrelated, pre-existing failures (configuration.test.ts, template-generation.test.ts) caused by EPERM when creating symlinks on this Windows host. These fail identically on clean main and are unrelated to this change.

• npm run format:check flags only package.json, a pre-existing local core.autocrlf worktree artifact (git ls-files --eol reports i/lf w/crlf); git treats the file as unchanged. All changed files pass Prettier.

• Tests added/updated where applicable

• Existing relevant tests pass

• Lint/type/build checks pass where available

Security Impact

• No security-sensitive behavior is changed
• Security-sensitive behavior is changed and explained below

Security notes:

• The harness only reads project-controlled bundled templates and writes generated output into isolated OS temporary directories that are cleaned up deterministically.
• No child processes, shell execution, dependency installation, Git operations, network/RPC calls, credentials, or configuration parsing are involved.
• No new runtime dependencies were introduced.

Release Impact

• No published/user-visible behavior change

• Patch

• Minor

• Breaking/pre-1.0 contract change

• Changeset included where required

• Changeset not required; reason explained below

Release notes:
Test-only and documentation changes. No shipped CLI behavior, command surface, template output, or package metadata is modified, so no changeset is required per docs/contributing/changesets.md.

Documentation

• Documentation updated where applicable
• No documentation change required

Updated docs/contributing/testing.md with a Performance Benchmarks section describing the harness, its offline/DEP-free scope, how to run it, how to raise iterations, and how to use it for regression comparison.

Checklist

• I read docs/contributing/wave-architecture.md for Stellar Wave work
• I stayed within the agreed issue scope
• I checked for overlapping open contributor PRs
• I did not include secrets or sensitive data
• I reviewed new/updated dependencies if any
• I considered failure/error paths, not only the happy path
• I am not disclosing an unresolved vulnerability publicly

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: f8683d5e-ef9b-4d43-a125-d307a530e559

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}