Repository navigation
Update SOC Handbook cost workbook and generate V3 package - #15452
Dr Bill Mcilhargey (billmcilhargey) wants to merge 14 commits into
Conversation
…and resource selectors
Reverts Workbooks/SentinelCosts.json and Workbooks/WorkbooksMetadata.json to master. All enhancements now live in the SOC Handbook solution workbook (AzureSentinelCost.json), bumped to 3.0.8. Structure: reorganized into 7 collapsible groups - Costing, Free Sentinel Data, E5 Benefits, Defender for Cloud Plan 2, Retention, Commitment Tiers, Data Lake. Parameters scoped per section; TimeRange/Workspace/Price remain global. Removed 10 hidden legacy placeholder items. New sections: Free Sentinel Data (documented free tables, savings), Retention (>90 day billable retention costs), Commitment Tiers (auto-detected current tier plus usage-based suggestion), Data Lake (per-table storage cost from mirrored volume at documented 6:1 compression, plus manual-entry table for ingestion/processing/query/insights meters that are not observable from the workspace). Behavior fixes: charts now derive bin size from the selected TimeRange (5m to 30d) instead of fixed daily bins; E5 and Defender P2 grant lines are prorated per interval rather than compared against a full-day allowance; commitment tier recommendation is withheld with a days-remaining countdown until 31 days of billable history exist; Logic App selector now emits concrete resource IDs (selected = Rank <= 10) instead of the value::all sentinel, which prevented the Azure Monitor metrics control from resolving scope. Defect fixes in the existing workbook: removed 85 invisible U+202F characters embedded in KQL; resolved duplicate item name 'EstimateMdcPlan2Discount - Copy'; corrected 'Defedner' typo; replaced hardcoded \ Defender P2 license price with a DfSP2LicensePrice parameter; scoped the licensing savings query to TimeRange instead of scanning all data; set fallbackResourceIds to [] per workbook guidance; stripped en-us locale codes from 11 Microsoft documentation URLs. Not included: Package/ artifacts are still at 3.0.7 and require regeneration via build-and-validate.ps1, which needs PowerShell 7 (only 5.1 is available locally).
Generate the requested V3 package candidate, synchronize solution release 3.0.9 and workbook metadata 1.6.0, and correct the release note. Runtime deployment evidence and the reproduced upstream ARM-TTK blank-array failure remain explicitly outstanding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
v-rusraut, please re-review the corrections requested in #14998. GitHub rejected reopening that PR, so this is the linked replacement (also consolidating the superseded #15020). Corrected: the V3 package outputs and ZIP are included; solution manifest, release notes and package are aligned at 3.0.9; workbook metadata/package content are aligned at 1.6.0; the description now covers every modified file. Focused workbook tests pass 15/15, the actual metadata validates, and both archived templates match their generated outputs byte-for-byte. Still outstanding: a genuine screenshot showing the deployed workbook loading actual data is not available from this environment, and full ARM-TTK acceptance is blocked by 12 blank-array findings in unchanged IncidentOverview/SentinelCentral content. All 12 findings reproduce at identical locations on an isolated unmodified upstream checkout ( Following your previous recommendation, this is a draft rather than an unsupported ready-for-review claim. Please review the supplied packaging/scope/version corrections and advise on the upstream validation findings. The deployment screenshot remains an explicit prerequisite before marking the contribution ready. CI infrastructure follow-upThe YAML validation workflow failed before inspecting content because its unauthenticated GitHub client hit HTTP 403 / API rate limit exceeded. I attempted a failed-job rerun, but GitHub requires repository administrator rights. Please rerun that workflow when possible. This is an additional explicit infrastructure blocker, not a YAML change or a passing-check claim. |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved grant, retention, and ingestion calculations can produce incorrect cost estimates.
6 open issues
Use Price when calculating Other category cost · New Normalize retained size by each bucket's covered duration · New Cap eligible ingestion per UTC day before summing usage · New Include AAD risk tables in summary eligibility queries · New Remove unused DfSP2LicensePrice input · New Use an independent historical context for retention scans · New
What changed in this PR
This follow-up updates the SOC Handbook cost workbook contribution and supplies the requested V3 package, aligning solution release versions while retaining the expanded cost-analysis sections.
Changes:
- Synchronizes the solution manifest and release records at 3.0.9.
- Increments AzureSentinelCost workbook metadata to 1.6.0.
- Supplies generated V3 content; deployment evidence and packaging acceptance remain outstanding.
| File | Description |
|---|---|
Workbooks/WorkbooksMetadata.json |
Bumps AzureSentinelCost metadata to 1.6.0. |
Solutions/SOC Handbook/ReleaseNotes.md |
Records workbook enhancements and the 3.0.9 packaging update. |
Solutions/SOC Handbook/Package/mainTemplate.json |
Generated V3 template; excluded from code review by repository guidance. |
Solutions/SOC Handbook/Data/Solution_SOC Handbook.json |
Updates the solution version to 3.0.9. |
Review effort: Balanced
Correct configurable ingestion pricing, covered Data Lake intervals, UTC daily grant caps and consistent M365 eligibility. Remove the unused licensing input and separate historical retention scans from the selected cost period. Add 37 offline regressions and regenerate the existing V3 3.0.9 package at workbook version 1.6.0; deployment evidence remains explicitly outstanding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
v-rusraut, please re-review the updated implementation and V3 package in commit 477fef3. All six automated review findings are corrected: selected-price handling, Data Lake interval normalization, UTC daily grant caps, Microsoft 365 risk-table eligibility parity, removal of the unused licensing input, and independent historical retention scans. The archive has been regenerated at the same unreleased 3.0.9 / workbook 1.6.0 versions and verified to contain all 37 corrected queries and the updated controls. Evidence: 37/37 new offline regressions (32 failed before correction), 15/15 existing workbook tests, metadata validation, and 72 Microsoft Kusto.Language semantic checks across all 36 Log Analytics queries with populated/unset-license fixtures, with zero semantic errors. The new tests are included in this PR. The exact seven-file scope, ZIP SHA-256 and local validation limitations are in the updated description. The local ARM-TTK finding still reproduces exactly on unchanged upstream; previous-head upstream ARM-TTK passed and the new head is being checked afresh. I have not fabricated the requested deployment screenshot or claimed live execution. That evidence still needs an authenticated approved non-production workspace/browser session, so this remains a draft while I complete that final environment-dependent requirement. Please review the supplied code and package corrections in the meantime. |
There was a problem hiding this comment.
🟡 Changes recommended
Four daily-rate calculations understate ingestion, costs, and savings for supported sub-day selections.
1 open issue
6 resolved since last review
Use an independent historical context for retention scans Remove unused DfSP2LicensePrice input Include AAD risk tables in summary eligibility queries Cap eligible ingestion per UTC day before summing usage Normalize retained size by each bucket's covered duration Use Price when calculating Other category cost
Review effort: Balanced
Use actual selected duration for workspace daily ingestion/cost and free-data daily ingestion/savings. Add four reviewer-example regressions (6 GB over six hours is 24 GB/day), preserve the 3.0.9 candidate version, and regenerate the V3 package. All 41 workbook regressions and 72 Microsoft KQL semantic checks pass; live deployment evidence remains outstanding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
v-rusraut, please re-review the latest code/package update at 1216049, following 477fef3 and 070361e. All eight current automated review findings are corrected, including the later sub-day daily-rate and inclusive-end findings. The selected interval is now consistent across all affected ingestion, cost, free-data and Defender views; historical probes retain their independent scope. The V3 candidate is freshly regenerated at 3.0.9 / workbook 1.6.0, with all 37 packaged queries matching final source. Verification: 44/44 offline regressions, 15/15 existing workbook tests, metadata validation and 72 Microsoft Kusto.Language semantic checks with zero errors. The reviewer's six-hour rate example and next-midnight endpoint example both reproduced the original errors before correction and now pass. All addressed threads have concrete fix replies. The preceding correction heads completed upstream checks successfully, including YAML and ARM-TTK. This update triggers fresh checks. The unchanged local ARM-TTK baseline limitation remains documented rather than suppressed. The outstanding environment-dependent deliverable is the requested screenshot showing the deployed corrected workbook loading actual lab data. No authenticated approved lab target has been supplied, so no Azure write or fabricated runtime screenshot was made. This remains a draft pending that genuine deployment evidence; please review the completed code and package corrections in the meantime. |
There was a problem hiding this comment.
🟡 Changes recommended
Defender ingestion views still disagree with benefit calculations at the selected end boundary, and required live-data deployment evidence remains outstanding.
1 open issue
1 resolved since last review
Review effort: Balanced
Use an exclusive selected end consistently across ingestion, costs, free-data summaries/trends and Defender eligibility/node views. Add midnight-boundary regressions and a workbook-wide predicate guard; document the interval convention and regenerate the existing V3 3.0.9 candidate. All 44 regressions, 15 existing workbook tests and 72 Microsoft KQL semantic checks pass. Live deployment evidence remains explicitly outstanding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use hourly-or-coarser bins and apportion overlapping source hours for storage and grant estimates. Add hourly-source regressions and regenerate the existing V3 3.0.9 candidate. Live deployment evidence and reproduced upstream ARM-TTK findings remain outstanding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
v-skharode, this PR is ready for your review.
The captures retain the visible empty/no-data states in some panels; the existing validation limitations documented in the PR description still apply. Please let me know if anything else is needed for review or validation. Thanks! Workbook screenshots (all 13, in workbook order)1. Workbook overview and workspace ingestion summary2. Workspace ingestion breakdown and trend3. Data lake scope, assumptions, and billing limitations4. Data lake parameters, estimates, and table breakdown5. Retained-volume empty state and free data summary6. Free data breakdown and trend7. Microsoft 365 benefit eligibility, seats, and summary8. Microsoft 365 eligible tables and grant-comparison empty state9. Defender for Servers Plan 2 benefit, protected nodes, and summary10. Defender for Servers eligible tables and grant-comparison empty state11. Retention summary and historical-data empty state12. Commitment tiers13. SOAR summary and Logic Apps inventory |
Dr Bill Mcilhargey (billmcilhargey)
left a comment
There was a problem hiding this comment.
Follow-up review — 29 October 2026
Reviewed current head 2c567bb29a1f4ed2d09f2a79c6de86360cc87b4d, the seven-file PR scope, all inline feedback, and the review/discussion history. All nine existing inline threads are already resolved and have corresponding fixes and replies. I found no additional high-confidence defect requiring a source change, so I have not created an unnecessary commit or regenerated the unchanged release candidate.
Fresh verification
- 73/73 Pester calculation regressions passed, with zero skipped tests.
- 15/15 existing workbook-template/metadata validator unit tests passed. The local harness reports that GitHub App authentication/PR context is unavailable; these unit-test results are not a claim that authenticated PR-diff checks or live Azure queries ran locally.
- All 37 packaged visualization queries exactly match source. All six query-owned time contexts (three hourly-overlap trends and three retention views) also match, and the removed licensing-price input is absent from both notebooks.
- Both ZIP entries,
mainTemplate.jsonandcreateUiDefinition.json, match their generated files byte-for-byte. ZIP SHA-256 remainsABC6C964F3EBA8D2304B53AC22C019480D5B9A4765494584D11DAB225BAA5CD2. - Solution/workbook versions remain 3.0.9 / 1.6.0, and the reviewed source changes pass
git diff --check. - Current GitHub checks report no failures; skipped/neutral checks are not counted as passes. A human approval is still required.
Outstanding review action
The latest Copilot review on this head failed on 28 October rather than returning a new finding. I attempted to request another review, but GitHub rejects the API request because the authenticated account lacks upstream write/reviewer-request permissions. The available browser is not signed in, so I could not retry through the PR UI either. A maintainer should re-request the Copilot review.
No new Azure deployment or live query execution was performed. The existing screenshot/no-data limitations remain applicable; these offline checks do not establish that every panel loads live data.
Regenerate the existing V3 candidate and refresh its release note. All 37 cost-workbook queries and both archive entries match source; 88 focused tests pass. Full local validation still reports the 12 pre-existing upstream ARM-TTK blank-property findings, with no validator changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Repackaged the corrected SOC Handbook solution and pushed commit d843b1f. Download the freshly rebuilt V3 3.0.9 package
Local acceptance is still blocked: the packager's ARM-TTK check and the independent canonical full validation runner both reproduce The PR description now points to the new archive and checksum, preserves all 13 screenshot links, and clearly separates prior-head CI evidence from this new head's checks. Fresh GitHub checks have been triggered; their results are not assumed from the old head. No Azure deployment was performed. |
Exclude billable records from all six free-data panels and calculate commitment-tier volume and readiness from hourly Usage aggregates. Add regression coverage and regenerate the existing SOC Handbook 3.0.9 candidate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dr Bill Mcilhargey (billmcilhargey)
left a comment
There was a problem hiding this comment.
Review recommendations addressed
Pushed e4d59c8, addressing both outstanding recommendations:
- All six free-data volume/savings queries exclude billable Usage records before aggregation. The reviewer's mixed SecurityAlert example is covered by regressions.
- Commitment-tier volume and history readiness now use hourly billable Usage aggregates instead of
union *over raw events. MB-to-GB conversion, the 31-day gate, existing tier-selection/output logic, and independence from the global time selection are covered.
Workbook guidance and the existing release-note row were updated. The same unreleased SOC Handbook 3.0.9 / AzureSentinelCost 1.6.0 candidate was regenerated; no unrelated content or protected validator/tooling implementation was changed.
Verification
- 90/90 Pester calculation regressions passed, including 17 new cases; the original 73 remain passing.
- 15/15 existing workbook-template/metadata validator unit tests passed. These unit tests do not establish authenticated PR-diff or live Azure validation.
- All 13 packaged workbooks, 37 cost-workbook visualization queries, and 7 query-owned historical/trend contexts were verified; both archived templates match their generated files byte-for-byte.
git diff --checkpasses, and the pushed branch matches the verified commit.
Download the updated V3 3.0.9 package — this supersedes the earlier candidate download links in this discussion.
SHA-256: 30F1E675A7EE8435A8B971DDF1A2820B660F659361DE0A81B2EFA20295A8521D.
Acceptance limitation retained
The canonical full local validation reports 9 validators passed / 1 failed / 11 skipped and explicitly marks the package not accepted. ARM-TTK has 49 checks passed / 1 failed (Template Should Not Contain Blanks), reporting the same 12 blank-property locations documented previously. Both affected generated payload lines (750 and 1318) are byte-for-byte unchanged from the previous head. No validator was weakened and no unrelated workbook was changed to suppress this failure.
The local KQL validator does not cover these workbook queries, and no new live Azure queries or deployment were performed. Fresh-head GitHub CI and maintainer acceptance are separate from these offline results.
There was a problem hiding this comment.
🟡 Changes recommended
The query still recommends a concrete commitment tier from combined ingestion across multiple workspaces, which can mislead users even when the status is 'Not evaluated'.
1 open issue
2 resolved since last review
Review effort: Balanced
| "type": 3, | ||
| "content": { | ||
| "version": "KqlItem/1.0", | ||
| "query": "let reviewDays = 31.0; let reviewStart = ago(31d); let historyProbeStart = ago(32d); let currentTier = '{CurrentCommitmentTier}'; let tierLevels = dynamic([50,100,200,300,400,500,1000,2000,5000]); Usage | where TimeGenerated >= historyProbeStart and TimeGenerated < now() | where IsBillable == true | summarize TotalBilledMB = sumif(Quantity, TimeGenerated >= reviewStart), FirstObserved = min(TimeGenerated) | extend ObservedHistoryDays = iif(isnull(FirstObserved), 0.0, min_of(reviewDays, datetime_diff('second', now(), FirstObserved) / 86400.0)) | extend DaysRemaining = toint(ceiling(max_of(reviewDays - ObservedHistoryDays, 0.0))) | extend AvgDailyGB = iif(DaysRemaining == 0, TotalBilledMB / 1024.0 / reviewDays, real(null)), TierLevels = tierLevels | mv-expand TierLevel = TierLevels to typeof(int) | extend IsEligible = TierLevel >= AvgDailyGB | summarize SuggestedTierGBPerDay = minif(TierLevel, IsEligible), AvgDailyGB = any(AvgDailyGB), ObservedHistoryDays = any(ObservedHistoryDays), DaysRemaining = any(DaysRemaining) | extend suggestedTier = case(DaysRemaining > 0, strcat('Ready in ', tostring(DaysRemaining), iif(DaysRemaining == 1, ' day', ' days')), AvgDailyGB < 50.0, 'Pay-As-You-Go', isnotempty(SuggestedTierGBPerDay), strcat(tostring(SuggestedTierGBPerDay), ' GB/day commitment'), 'Above 5000 GB/day - review Azure portal options') | extend alignmentStatus = case(DaysRemaining > 0, 'Not Ready', currentTier == 'Not available' or currentTier == 'Unknown' or currentTier == 'Multiple workspaces', 'Not evaluated', suggestedTier startswith 'Above 5000 GB/day', 'Review in portal', currentTier == suggestedTier, 'Aligned', 'Review tier') | project Current = currentTier, Suggested = suggestedTier, Status = alignmentStatus, ['Avg daily GB'] = round(AvgDailyGB, 2)", |














Repackaged release candidate (29 October 2026)
Rebuilt the existing SOC Handbook 3.0.9 / AzureSentinelCost 1.6.0 legacy V3 candidate from the reviewed source in commit
d843b1f8efcf57582ecc2f7c60e390291899b492; no additional release increment or new workbook behavior was introduced.157149C6AED64FF21E453E5B9199A5470B23261824AEDC63041239DC9CD41065.Proposed Changes
Follow up on #14998 and its superseded duplicate #15020. GitHub rejected reopening the original PR, so this replacement includes the original SOC Handbook cost workbook contribution, the requested V3 package, and all nine subsequent automated review corrections.
The requested 13 workbook screenshots are embedded below, captioned and arranged in workbook section order, from workspace ingestion through the SOAR summary. The requested V3 solution package is also included, with a direct download link. This PR is ready for review.
Review Corrections
Correction commits:
477fef3cc3de1266fdc39d43b2b455d5ddef821a,070361eda8bd28adb1efdb76814e22cd58aec715,121604912085af24f9232368abbe4a1109916eaa, and2c567bb29a1f4ed2d09f2a79c6de86360cc87b4d.Exact Changed-File Scope
Solutions/SOC Handbook/Workbooks/AzureSentinelCost.jsonSolutions/SOC Handbook/Data/Solution_SOC Handbook.jsonSolutions/SOC Handbook/ReleaseNotes.mdSolutions/SOC Handbook/Tests/AzureSentinelCost.Tests.ps1Solutions/SOC Handbook/Package/mainTemplate.jsonSolutions/SOC Handbook/Package/3.0.9.zipWorkbooks/WorkbooksMetadata.json- only AzureSentinelCost's metadata version and publish date.The standalone SentinelCosts workbook is unchanged. Generated createUiDefinition.json and testParameters.json remain unchanged from the base. No shared validators, workflows, or unrelated workbooks were modified. Screenshot assets are hosted on a separate branch in the author's fork and do not add files to this PR's workbook or package scope.
Packaging and Versions
Download the V3 solution package 3.0.9 (.zip)
The requested package has been created and committed. The canonical Sentinel-only V3 tool regenerated this same unreleased candidate in documented catalog mode; no additional release increment was introduced. The download link above was checked against the recorded ZIP SHA-256.
157149C6AED64FF21E453E5B9199A5470B23261824AEDC63041239DC9CD41065.Recorded Local Testing
These are the earlier implementation-validation results. Fresh 29 October repackaging and test results are recorded above; source workbook queries and test definitions remain unchanged.
Run the regressions from the repository root with PowerShell 7 and Pester 5+:
Prior-Head CI Status (28 October 2026)
Checked at 2026-10-28 15:38 UTC for head
2c567bb29a1f4ed2d09f2a79c6de86360cc87b4d:View the PR checks.
Workbook Screenshots
Here are the requested author-provided screenshots for validation, embedded directly in the description and arranged in the workbook's top-to-bottom order. The visible global selection is Last 7 days; retention and commitment-tier views retain their documented independent history windows.
The captures show populated summary tiles and source-table breakdowns, together with the visible empty/no-data states. The retained-volume chart and both grant-comparison charts show no-data messages despite populated related summaries, while the retention panel reports no billable data older than 90 days. The screenshots are visual validation evidence, not proof that every panel returned populated results. No new Azure deployment or live query execution was performed for this description refresh.
1. Workbook overview and workspace ingestion summary
2. Workspace ingestion breakdown and trend
3. Data lake scope, assumptions, and billing limitations
4. Data lake parameters, estimates, and table breakdown
5. Retained-volume empty state and free data summary
6. Free data breakdown and trend
7. Microsoft 365 benefit eligibility, seats, and summary
8. Microsoft 365 eligible tables and grant-comparison empty state
9. Defender for Servers Plan 2 benefit, protected nodes, and summary
10. Defender for Servers eligible tables and grant-comparison empty state
11. Retention summary and historical-data empty state
12. Commitment tiers
13. SOAR summary and Logic Apps inventory
Earlier Local-Run Limitations
These notes describe the earlier local environment and are retained for traceability; they must not be read as unresolved failures of the now-completed current-head CI checks.
cc043cefcbbaseline in the existing IncidentOverview/SentinelCentral payloads. Current-head GitHub ARM-TTK checks subsequently passed. No gate was bypassed, no validators were weakened, and no unrelated workbooks were modified.Review Request
v-rusraut, the requested V3 package, all nine code-review corrections, and the requested workbook screenshots for validation are now supplied. Current-head CI has completed with no reported failures or jobs in progress. Please re-review the implementation, the linked package, and the 13 screenshots embedded above. Please let me know if anything else is needed for review or validation.