SentinelToXDRMigration: qualification follow-ups, XDR static KQL override, and NPF metrics

.claude/skills/sentinel-xdr-migration/SKILL.md

@@ -10,10 +10,11 @@ Read and follow:
 `Tools\SentinelToXDRMigration\agent-workflows\end-to-end-migration.md`
 
 Use `sentinel-xdr-migration` and the repository PowerShell packaging entry
-points for all deterministic actions. Default to `authoring`; ask before
-optional `qualification`. Use V3.1 through `package-v3-1` for Sentinel and XDR
-solution packaging, resume `workflow-state.json`, preserve disabled rules, and
-obtain explicit approval before environment writes.
+points for all deterministic actions. Default to `qualification` (with
+`--workflow-profile authoring` available as an explicit offline/ISV fallback).
+Use V3.1 through `package-v3-1` for Sentinel and XDR solution packaging, resume
+`workflow-state.json`, preserve disabled rules, and obtain explicit approval
+before environment writes.
 
 The goal is solution migration and validated publication. Follow the
 canonical workflow for the strict over-50% partial-publication policy,

.cursor/rules/sentinel-xdr-migration.mdc

@@ -12,7 +12,7 @@ packaging, or AR/CD qualification, read and follow
 `Tools/SentinelToXDRMigration/agent-workflows/end-to-end-migration.md`.
 
 That document is canonical. Use repository CLIs for deterministic behavior,
-default to the authoring profile, and ask before optional qualification.
-Qualification deployment, mock ingestion, and parity are not publishing
-requirements. Use V3.1 through `package-v3-1` for Sentinel and XDR solution
+default to the `qualification` profile, and use `--workflow-profile authoring`
+as an explicit offline/ISV fallback when no live lab is available.
+Use V3.1 through `package-v3-1` for Sentinel and XDR solution
 packages. Resume persisted workflow state and require approval before writes.

.github/agents/xdr-solution-manager.agent.md

@@ -131,16 +131,17 @@ review, runtime/schema, packaging, authentication, or write gates.
 
 ## Migration workflow
 
-1. Run `workflow-status` first. If no state exists, default to **Authoring**
-   and ask before optional **Qualification** (or **Cancel**). Explain that
-   Authoring stops after packaging/reporting while Qualification adds
-   deployment, mock ingestion, and alert parity.
-2. Initialize or resume with the selected profile (defaults to `authoring`):
+1. Run `workflow-status` first. If no state exists, default to **Qualification**
+   (or allow explicit **Authoring** as an offline/ISV fallback when no live lab
+   is available, or **Cancel**). Explain that Qualification adds deployment,
+   mock ingestion, and alert parity while Authoring stops after
+   packaging/reporting.
+2. Initialize or resume with the selected profile (defaults to `qualification`):
 
    ```powershell
    sentinel-xdr-migration workflow-init `
      --solution "<solution-path>" `
-     --workflow-profile "<authoring-or-qualification>" `
+     --workflow-profile "<qualification-or-authoring>" `
      --tenant-id "<tenant-guid>" `
      --subscription-id "<subscription-guid>" `
      --workspace-resource-id "<workspace-arm-id>" `

.github/skills/sentinel-xdr-end-to-end-orchestrator/SKILL.md

@@ -49,8 +49,9 @@ the ARM ID again. Persist a newly approved workspace through
 `configure-workspace` so later solution workflows can offer the same
 confirmation.
 
-Default to **Authoring** and ask before optional **Qualification** through
-structured elicitation. If selection is cancelled, do not create workflow state.
+Default to **Qualification** (with **Authoring** available as an explicit
+offline/ISV fallback when no live lab is available) through structured
+elicitation. If selection is cancelled, do not create workflow state.
 A test tenant or workspace does not imply qualification consent. Require
 explicit approval immediately before deployment or ingestion writes.
 

.github/workflows/npf-pr-labeler.yaml

@@ -44,16 +44,138 @@ jobs:
               }
             }
 
-            const hasXdrContent = activePaths.some(p =>
-              /^Solutions\/[^/]+\/(XDR Detections|XDR Workbooks)\//.test(p)
+            const existing = new Set(
+              (context.payload.pull_request?.labels || []).map(l => l.name)
             );
+            const headSha = context.payload.pull_request?.head?.sha;
+            const baseSha = context.payload.pull_request?.base?.sha;
+
+            async function fetchFileText(repoPath) {
+              for (const ref of [headSha, baseSha].filter(Boolean)) {
+                try {
+                  const resp = await github.rest.repos.getContent({
+                    owner: context.repo.owner,
+                    repo: context.repo.repo,
+                    path: repoPath,
+                    ref
+                  });
+                  if (resp.data && resp.data.content) {
+                    return Buffer.from(resp.data.content, 'base64').toString('utf8');
+                  }
+                } catch (_) {
+                  // Try fallback ref
+                }
+              }
+              return null;
+            }
+
+            async function pathExistsAtHead(repoPath) {
+              for (const ref of [headSha, baseSha].filter(Boolean)) {
+                try {
+                  const resp = await github.rest.repos.getContent({
+                    owner: context.repo.owner,
+                    repo: context.repo.repo,
+                    path: repoPath,
+                    ref
+                  });
+                  if (resp.data) {
+                    return true;
+                  }
+                } catch (_) {
+                  // Try fallback ref
+                }
+              }
+              return false;
+            }
+
+            let hasXdrContent = activePaths.some(p =>
+              /^Solutions\/[^/]+\/(XDR Detections|XDR Workbooks)\//.test(p)
+            ) || existing.has('XDR');
+
+            if (!hasXdrContent) {
+              for (const sol of solutionNames) {
+                const touchesPackage = activePaths.some(
+                  p => p === `Solutions/${sol}/Package/mainTemplate.json` || p.startsWith(`Solutions/${sol}/Package/`)
+                );
+                if (touchesPackage) {
+                  const hasXdrDet = await pathExistsAtHead(`Solutions/${sol}/XDR Detections`);
+                  const hasXdrWb = !hasXdrDet && (await pathExistsAtHead(`Solutions/${sol}/XDR Workbooks`));
+                  if (hasXdrDet || hasXdrWb) {
+                    hasXdrContent = true;
+                    break;
+                  }
+                }
+              }
+            }
+
             const hasValidationEvidence = activePaths.some(p =>
               /^Solutions\/[^/]+\/Evidence\/(pr-qualification\/(pr-qualification-result\.json|qualification-report\.html|qualification-summary\.md)|workflow-state\.json|decision-manifest\.json|[^/]+\/evidence\.json)$/.test(p)
             );
-            const hasQualificationEvidence = activePaths.some(p =>
+            let hasQualificationEvidence = activePaths.some(p =>
               /^Solutions\/[^/]+\/Evidence\/(qualification-evidence\.json|receipts\/(alert-parity-live|workbook-queries-live|arm-deployment-readback|graph-detection-deployment)\.json)$/.test(p)
             );
 
+            const verdicts = {};
+            for (const sol of [...solutionNames].sort()) {
+              const qualResultCandidates = activePaths.filter(
+                p =>
+                  p === `Solutions/${sol}/Evidence/pr-qualification/pr-qualification-result.json` ||
+                  p === `Solutions/${sol}/Evidence/pr-qualification-result.json`
+              );
+              let solVerdict = null;
+              for (const cand of qualResultCandidates) {
+                const rawJson = await fetchFileText(cand);
+                if (!rawJson) continue;
+                try {
+                  const parsed = JSON.parse(rawJson.replace(/^\uFEFF/, ''));
+                  const modeVal = String(parsed?.mode || parsed?.workflowProfile || '').trim().toLowerCase();
+                  const qualStatus = String(parsed?.qualificationStatus || '').trim();
+                  const verdictVal = String(parsed?.verdict || parsed?.overallVerdict || '').trim();
+                  if (
+                    modeVal === 'qualification' ||
+                    modeVal === 'full' ||
+                    qualStatus === 'Qualified' ||
+                    qualStatus === 'Conditionally Qualified' ||
+                    verdictVal === 'Qualified' ||
+                    verdictVal === 'Conditionally Qualified'
+                  ) {
+                    hasQualificationEvidence = true;
+                  }
+                  if (verdictVal) {
+                    solVerdict = verdictVal;
+                    break;
+                  }
+                } catch (_) {
+                  // Ignore parse errors
+                }
+              }
+
+              const summaryCandidates = activePaths.filter(
+                p =>
+                  p === `Solutions/${sol}/Evidence/pr-qualification/qualification-summary.md` ||
+                  p === `Solutions/${sol}/Evidence/qualification-summary.md`
+              );
+              for (const cand of summaryCandidates) {
+                const mdText = await fetchFileText(cand);
+                if (!mdText) continue;
+                if (/\bConditionally Qualified\b/i.test(mdText)) {
+                  hasQualificationEvidence = true;
+                  if (!solVerdict) solVerdict = 'Conditionally Qualified';
+                } else if (/\bQualified\b/i.test(mdText)) {
+                  hasQualificationEvidence = true;
+                  if (!solVerdict) solVerdict = 'Qualified';
+                } else if (!solVerdict && /\bPartially Validated\b/i.test(mdText)) {
+                  solVerdict = 'Partially Validated';
+                } else if (!solVerdict && /\bValidated\b/i.test(mdText)) {
+                  solVerdict = 'Validated';
+                }
+              }
+
+              if (solVerdict) {
+                verdicts[sol] = solVerdict;
+              }
+            }
+
             const desired = new Set();
             const npfModes = [];
             if (solutionNames.size > 0) {
@@ -74,30 +196,12 @@ jobs:
             }
 
             // Determine 1P Solution / ISV Solution / Community Solution from SolutionMetadata.json
-            const headSha = context.payload.pull_request?.head?.sha;
-            const baseSha = context.payload.pull_request?.base?.sha;
             const publisherTiers = new Set();
             const solutionDetails = [];
 
             for (const sol of [...solutionNames].sort()) {
               const metaPath = `Solutions/${sol}/SolutionMetadata.json`;
-              let metaText = null;
-              for (const ref of [headSha, baseSha].filter(Boolean)) {
-                try {
-                  const resp = await github.rest.repos.getContent({
-                    owner: context.repo.owner,
-                    repo: context.repo.repo,
-                    path: metaPath,
-                    ref
-                  });
-                  if (resp.data && resp.data.content) {
-                    metaText = Buffer.from(resp.data.content, 'base64').toString('utf8');
-                    break;
-                  }
-                } catch (_) {
-                  // Try fallback ref
-                }
-              }
+              const metaText = await fetchFileText(metaPath);
               let tierLabel = 'Unknown';
               if (metaText) {
                 try {
@@ -127,9 +231,6 @@ jobs:
               return;
             }
 
-            const existing = new Set(
-              (context.payload.pull_request?.labels || []).map(l => l.name)
-            );
             const labelsToAdd = [...desired].filter(l => !existing.has(l));
 
             if (labelsToAdd.length > 0) {
@@ -144,17 +245,61 @@ jobs:
               core.info(`All applicable labels already present: ${[...desired].join(', ')}`);
             }
 
+            if (npfModes.length === 0) {
+              core.info('PR has no NPF modes (Migration/Validation/Qualification); skipping npf-meta block.');
+              return;
+            }
+
+            const highestStage = npfModes.includes('Qualification')
+              ? 'Qualification'
+              : (npfModes.includes('Validation') ? 'Validation' : 'Migration');
+
+            let commitPasses = Number(context.payload.pull_request?.commits || 0);
+            if (!commitPasses || commitPasses < 1) {
+              try {
+                const prCommits = await github.paginate(github.rest.pulls.listCommits, {
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  pull_number: context.issue.number,
+                  per_page: 100
+                });
+                commitPasses = Math.max(1, prCommits.length);
+              } catch (_) {
+                commitPasses = 1;
+              }
+            }
+            const solCount = Math.max(1, solutionDetails.length);
+            const passesPerSolution = Number((commitPasses / solCount).toFixed(2));
+
             // Build machine-readable npf-meta block for PR body and step summary
+            const currentBody = context.payload.pull_request?.body || '';
+            const metaRegex = /<!--\s*npf-meta:\s*(\{.*?\})\s*-->/s;
+            let priorSyncObserved = false;
+            const existingMetaMatch = currentBody.match(metaRegex);
+            if (existingMetaMatch) {
+              try {
+                const parsedMeta = JSON.parse(existingMetaMatch[1]);
+                if (parsedMeta && parsedMeta.synchronizePushObserved === true) {
+                  priorSyncObserved = true;
+                }
+              } catch (_err) {
+                // Ignore malformed prior metadata block
+              }
+            }
+            const synchronizePushObserved = priorSyncObserved || (context.payload.action === 'synchronize');
             const npfMeta = {
               solutionCount: solutionDetails.length,
               solutions: solutionDetails.map(s => s.name),
               publisherTiers: [...publisherTiers],
-              npfModes
+              npfModes,
+              highestStage,
+              commitPasses,
+              passesPerSolution,
+              verdicts,
+              synchronizePushObserved
             };
             const metaComment = `<!-- npf-meta: ${JSON.stringify(npfMeta)} -->`;
 
-            const currentBody = context.payload.pull_request?.body || '';
-            const metaRegex = /<!--\s*npf-meta:\s*\{.*?\}\s*-->/s;
             let updatedBody = currentBody;
             if (metaRegex.test(currentBody)) {
               updatedBody = currentBody.replace(metaRegex, metaComment);

.script/local-validation/build-and-validate.ps1

@@ -1034,8 +1034,8 @@ if (-not $SkipValidation) {
         if ($has80 -and (Test-Path $kqlProject)) {
             # Collect all YAML/JSON content files from the solution for scoping
             $kqlFiles = Get-SolutionContentFiles -SolPath $solutionPath `
-                -SubDirs @("Analytic Rules", "Hunting Queries", "Parsers", "Data Connectors") `
-                -Extensions @("*.yaml", "*.json")
+                -SubDirs @("Analytic Rules", "XDR Detections", "Hunting Queries", "Parsers", "Data Connectors") `
+                -Extensions @("*.yaml", "*.yml", "*.json")
 
             if ($kqlFiles.Count -gt 0) {
                 $kqlFilter = Build-DotnetTestFilter -Files $kqlFiles -UseEncodedPath
@@ -1124,9 +1124,9 @@ if (-not $SkipValidation) {
                 }
             }
             else {
-                $xdrKqlCheckFiles = Get-SolutionContentFiles -SolPath $solutionPath -SubDirs @("XDR Detections", "XDR Workbooks") -Extensions @("*.yaml", "*.yml", "*.json")
+                $xdrKqlCheckFiles = Get-SolutionContentFiles -SolPath $solutionPath -SubDirs @("XDR Workbooks") -Extensions @("*.json")
                 $noKqlReason = if ($xdrKqlCheckFiles.Count -gt 0) {
-                    "No legacy Sentinel KQL files found (.NET KQL validator does not support XDR Detections or XDR Workbooks)"
+                    "No Sentinel or XDR Detection KQL files found (.NET KQL validator does not support XDR Workbooks; validated via static XDR KQL validator)"
                 }
                 else {
                     "No content files found"