feat: match a customer-configured manifest filename

[pankajastro]

Summary

Approach changed since earlier commits/review — replaced the customer-configured env var with automatic discovery. No configuration needed.

Discovery previously matched only files literally named manifest.json. Now a file is treated as a manifest when its name contains "manifest" (case-insensitive) and its content validates as one - covering manifest.json, manifest_full.json, manifest_by_schedule.json, etc. automatically.

Every manifest found gets its own slim companion, named after itself:

manifest.json      -> manifest.slim.json   (unchanged default)
manifest_full.json -> manifest_full.slim.json

So a directory with multiple manifests (e.g. per-schedule manifests, each read by a different DAG) gets one slim file per manifest instead of one contested file - no picking a winner, no collision. Three dbt projects with two manifests each produce six slim files, one per manifest.

Sidecar schema bumped to v2 in response to review (a directory can describe more than one manifest now, so a single top-level hash/slim pointer isn't enough). .astro/dbt_metadata.json keeps its fixed name, but filtered_manifest is replaced by manifests, a map keyed by source filename, each with its own hash and slim pointer:

{
  "schema": 2,
  "version": {"algo": "sha256-tree-v2", "hash": "…"},
  "manifests": {
    "manifest_full.json": {
      "version": {"algo": "sha256-manifest-v2", "hash": "…"},
      "slim_manifest": {"schema": 1, "path": "manifest_full.slim.json", "version": {...}}
    },
    "manifest_by_schedule.json": { "...": "..." }
  }
}

A consumer looks up its own manifest's entry instead of trusting the top-level version/pointer when a directory has several. The top-level version is kept for backward compat and is deterministic (alphabetically-first manifest in the directory).

This also closed a real concurrency hazard flagged in review: multiple manifests sharing a directory are hashed/slimmed in parallel, then written in one grouped pass per directory (one writer, no race) - verified with a 300-iteration stress test at real parallelism.

Cleanup matches a slim file by its .slim.json suffix instead of one fixed name, so it can find and remove any of them; ownership is still checked per file via its own _generated_by marker before deleting.

Also fixed in review: a failure slimming one manifest in a project root no longer aborts the whole unit - the project's tree-hash sidecar and any already-slimmed sibling manifest are no longer discarded.

Follow-up tracked separately: BOSS-738 for the Cosmos Boost plugin repo to consume the new manifests map instead of a single pointer.

Testing

• go build/go vet/gofmt clean; full pkg/cosmosboost/... suite passing (golang:1.26 container - no local Go toolchain).
• Key tests: TestRunSharedDirectorySidecarDescribesBothManifests (schema v2 map + top-level version determinism across repeated runs), TestRunDoesNotFlagOtherDbtArtifactsAsManifests, TestRunSlimsEveryManifestInProjectRoot, TestRunProjectRootSlimFailureIsolatedToOneManifest, TestRunHandlesMultipleProjectsWithDifferentManifestNames, TestIsManifestCandidateName, TestSlimNameFor, TestCleanupRemovesCustomNamedSlimManifest, TestCleanupKeepsForeignSlimJSONSuffixedFile.
• E2E'd against a copy of a real customer directory holding two manifests (9275 vs 2300 nodes, each meant for a different DAG): both now get stamped with their own correctly-scoped slim file, no configuration needed.
• E2E'd the exact "3 projects x 2 manifests each" scenario: confirmed 6 slim files, one per manifest.
• Stress-tested the grouped-write path at GOMAXPROCS=8 across 300 iterations: zero corruption or lost updates in the sidecar.
• Verified EnsureClean removes a stale slim file after a manifest is renamed between deploys (no orphaned files left behind).

🤖 Generated with Claude Code

[coveralls-official]

Coverage Report for CI Build 37749960856

Coverage increased (+0.06%) to 44.913%

Details

• Coverage increased (+0.06%) from the base build.
• Patch coverage: 5 uncovered changes across 1 file (123 of 128 lines covered, 96.09%).
• 1 coverage regression across 1 file.

Uncovered Changes

File	Changed	Covered	%
pkg/cosmosboost/precompute/precompute.go	106	101	95.28%
Total (6 files)	128	123	96.09%

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
pkg/cosmosboost/precompute/precompute.go	1	97.17%

---

Coverage Stats

Relevant Lines:	60181
Covered Lines:	27029
Line Coverage:	44.91%
Coverage Strength:	9.95 hits per line

---

💛 - Coveralls

[pankajastro]

@kaxil heads up - the approach changed significantly since my earlier replies on this thread, so those are superseded.

Instead of a customer-configured env var (name override, then a JSON directory→filename map) plus an ambiguity guard that skipped stamping when a directory held more than one valid manifest, this now discovers manifests automatically: a file is treated as a manifest when its name contains "manifest" (case-insensitive) and its content validates as one - no configuration needed.

The bigger change: every manifest found now gets its own slim companion, named after itself (manifest_full.json → manifest_full.slim.json, manifest.json → manifest.slim.json as before). So the manifests_per_schedule/ case you originally flagged is no longer "skip the whole directory" - both manifests get correctly and separately stamped now, since there's no shared/fixed slim filename left to collide on.

PR description is updated to match. Would appreciate another look when you have time.

[tatiana]

Thanks, @pankajastro , for implementing the original approach we had agreed with.

I left two comments inline, I consider them blockers.

Please, could you confirm if we have a follow-up ticket for the plugin work?

[pankajastro]

Thanks, @pankajastro , for implementing the original approach we had agreed with.

I left two comments inline, I consider them blockers.

Please, could you confirm if we have a follow-up ticket for the plugin work?

Thanks for the review. Addressed your feedback and created a follow-up issue for plugin work

[pankajkoti]

Overall this rework is a good improvement. I left two inline comments (the schema bump, and a minor stale comment).

One more thing that is not in this diff so I could not inline it: the scaffold gitignore at pkg/airflowrt/include/gitignore:18 is still the literal **/.astro/manifest.slim.json. This PR now writes custom-named slim files like manifest_full.slim.json, which that line does not match, so they show up untracked. Same thing that was fixed once in #2242. Broadening it to **/.astro/*manifest*.slim.json covers all of them while still matching only the slim manifests this step writes.

[pankajastro]

Fixed the scaffold gitignore too - broadened to **/.astro/*manifest*.slim.json, same pattern as #2242.

[tatiana]

Thanks for addressing the feedback, @pankajastro !