Add /_skill-validation monitoring endpoint

[backspace]

In an early step of the @cardstack/base migration, the AI assistant was broken because skills imports had changed and there was no test suite failure, it’s more of an integration question.

This adds an endpoint that Checkly can poll, it reports whether each loaded skill’s tools load properly.

With this branch deployed to staging:

{
  "data": {
    "type": "skill-validation",
    "id": "https://realms-staging.stack.cards/skills/",
    "attributes": {
      "status": "pass",
      "skillsChecked": 41,
      "toolsChecked": 48,
      "failures": []
    }
  }
}

The corresponding Checkly check passed when it was deployed:

Here’s an example response with failures:

  {
    "data": {
      "type": "skill-validation",
      "id": "https://app.boxel.ai/skills/",
      "attributes": {
        "status": "fail",
        "skillsChecked": 12,
        "toolsChecked": 31,
        "failures": [
          {
            "skill": "https://app.boxel.ai/skills/Skill/boxel-development",
            "module": "@cardstack/boxel-host/commands/patch-card",
            "name": "default",
            "error": "encountered error loading module \"@cardstack/boxel-host/commands/patch-card\":
  404: Not Found"
          },
          {
            "skill": "https://app.boxel.ai/skills/writing-assistant.md",
            "module": "https://app.boxel.ai/skills/tools/rewrite",
            "name": "RewriteTool",
            "error": "module https://app.boxel.ai/skills/tools/rewrite has no export \"RewriteTool\""
          }
        ]
      }
    }
  }

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files  ±0      1 suites  ±0   3h 3m 23s ⏱️ +57s
3 486 tests ±0  3 471 ✅ ±0  15 💤 ±0  0 ❌ ±0 
3 505 runs  ±0  3 490 ✅ ±0  15 💤 ±0  0 ❌ ±0 

Results for commit fc85798. ± Comparison against earlier commit 8b2e697.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   11m 42s ⏱️ +18s
1 829 tests ±0  1 829 ✅ +2  0 💤 ±0  0 ❌  - 2 
1 908 runs  ±0  1 908 ✅ +2  0 💤 ±0  0 ❌  - 2 

Results for commit fc85798. ± Comparison against earlier commit 8b2e697.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c28187b96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Adds a new realm-server monitoring endpoint (/_skill-validation) that Checkly can poll to verify that all skills in a realm reference tool modules/exports that the deployed host can actually load (via the prerenderer), preventing “silent” integration breakages (e.g. module renames / export renames) that won’t show up as unit-test failures.

Changes:

• Introduces /_skill-validation realm-server route + handler that finds skill sources (legacy Skill cards + skill-markdown files) and validates referenced tool codeRefs by prerendering unique modules and checking export names.
• Extends the module prerender/model response contract with an optional exports?: string[] list, and populates it in the host /module route.
• Refactors monitoring auth into a shared async helper (isAuthorizedToViewMonitoring) and adds comprehensive realm-server endpoint tests for both failure and success cases.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/runtime-common/index.ts	Adds optional exports?: string[] to ModulePrerenderModel/ModuleRenderResponse contract for non-BaseDef export validation.
packages/runtime-common/constants.ts	Adds markdownDefRef to query for MarkdownDef file rows (skill-markdown discovery).
packages/realm-server/utils/monitoring.ts	Adds shared async isAuthorizedToViewMonitoring() used by monitoring endpoints.
packages/realm-server/handlers/handle-queue-status.ts	Switches to shared async monitoring auth helper (fixes token comparison correctness with async token derivation).
packages/realm-server/routes.ts	Registers new /_skill-validation endpoint.
packages/realm-server/handlers/handle-skill-validation.ts	Implements the monitoring endpoint: skill discovery + prerender-based module/export validation + JSON:API response.
packages/realm-server/tests/server-endpoints/skill-validation-test.ts	New tests covering auth requirements, bad params, module-load failures, missing-export failures, and private-realm success.
packages/realm-server/tests/server-endpoints/queue-status-test.ts	Fixes test to await the async monitoring token.
packages/realm-server/tests/index.ts	Adds the new skill-validation test file to the test suite.
packages/host/app/routes/module.ts	Adds exports: Object.keys(module) to ready module models so prerender responses can report export names.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.