From 96b16a3c8734e7739d65bfce449aa89a4de0f98a Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Fri, 11 Sep 2026 22:06:21 -0700 Subject: [PATCH 01/50] Automate Power BI release packaging and fix template generation bugs Power BI releases ship three files: PowerBI-kql.zip and PowerBI-storage.zip (PBIT templates) and PowerBI-demo.zip (demo PBIX files). Only the first two were generated, and the last two releases stalled figuring out where the scripts stopped. Build-PowerBI now prunes each report's model once and renders it twice: a PBIT template and a self-contained PBIP project in release/pbix. Because both come from the same prune, the template and the demo report can't drift, and the manual "remove unused queries" step is gone. Templates ship with the data source parameters nulled; the projects keep the demo values so demo reports still refresh. Package-PowerBI is a new resumable command that works out which steps are done, does the next one, and validates the saved PBIX files. Saving from Power BI Desktop is the only manual step left. Fixes found along the way: - Build-PowerBI was broken on dev. A case-duplicate column added to AdvisorReservationRecommendations in #1849 made the model fail to load, and the script reported errors but still exited 0 with truncated PBIT files. Tables no report ships are now dropped before the model loads, so an unused broken table can't block a release, and the script fails loudly. - The data source nulling regex expected an escaped quote that was never there, so every shipped template had the demo storage account and cluster prefilled. - PBI_QueryOrder was double JSON encoded and leaked literal [storage] and [kql] prefixes into the query pane order. - DiagramLayout was never pruned, so templates shipped diagram entries for tables the model no longer had. - The Governance report listed PolicyDefinitions under Tables, but it's a query, so it was silently dropped from the template. - Package-Toolkit -Build -CopyFiles deleted the Power BI ZIP files that -Build had just generated. - -ZipPBI never created PowerBI-demo.zip, and its cleanup called Compress-Archive on a directory, which nests every part one folder deep and produces an unreadable PBIX. Confirmed unused: v14 ships the raw PBIX files. Removed. Co-Authored-By: Claude Opus 5 --- .claude/commands/release.md | 19 +- .../-internal-only--release-checklist.md | 14 +- docs-mslearn/toolkit/changelog.md | 3 + docs-wiki/Build-and-test.md | 23 +- docs-wiki/Release-process.md | 34 +- src/scripts/Build-PowerBI.ps1 | 701 +++++++++++++++--- src/scripts/Package-PowerBI.ps1 | 365 +++++++++ src/scripts/Package-Toolkit.ps1 | 82 +- src/scripts/README.md | 70 +- 9 files changed, 1102 insertions(+), 209 deletions(-) create mode 100644 src/scripts/Package-PowerBI.ps1 diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 4243580cb..437de3337 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -148,13 +148,26 @@ pwsh -Command "./src/scripts/Package-Toolkit.ps1 -Build -CopyFiles" If it fails, show the error and ask whether to investigate or skip. +### Package Power BI + +Power BI packaging is resumable and reports its own state. Run: + +```bash +pwsh -Command "./src/scripts/Package-PowerBI.ps1" +``` + +Relay its status output verbatim — it says exactly which step is next. When it reports projects that still need to be saved, tell the user to run `Package-PowerBI.ps1 -Open`, save each project as PBIX (refresh, Save as PBIX in `release/pbix`, sensitivity "Public", end on the Get started page), then say "done" so you can rerun the command to validate and package. + +If validation fails, show the reported issues verbatim. Each one names the file and the fix; don't guess at causes. + +Do not proceed until all three files exist: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. + ### Manual steps reminder After packaging succeeds, inform the user of remaining manual steps documented in the release checklist issue (include a link): -1. Power BI packaging -2. Publish release -3. Publish announcements +1. Publish release +2. Publish announcements ### Final issue update diff --git a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md index 75c7f3599..7ad3d862c 100644 --- a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md +++ b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md @@ -110,15 +110,17 @@ Status icons: > _This step is optional, but can catch issues earlier. You can also add the `-Build` parameter to the publish command in the next step._ - Ensure all tests pass: `/src/scripts/Test-PowerShell -Unit -Integration` - [ ] Package all release files (except Power BI): `/src/scripts/Package-Toolkit.ps1 -Build -CopyFiles` script -- [ ] Package Power BI files - - [ ] Run `Package-Toolkit.ps1 -OpenPBI` script. +- [ ] Package Power BI files: `/src/scripts/Package-PowerBI.ps1` + > _This command is resumable. Run it, do what it asks, then run it again. It reports what's done, what's left, and checks the saved PBIX files for missed steps._ + - [ ] Run `Package-PowerBI.ps1 -Open` to open the projects that need to be saved. - [ ] Save and close each Power BI project: - - Select the `/release/pbix` folder. - - Change the file extension to PBIX. + - Refresh the report so demo data is loaded. + - Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. - When prompted, set the sensitivity to "Public". - - Manually remove unused queries based on what's documented in Build-PowerBI.ps1 ~line 120. - Verify all pages, switch to the Get started page, and save again. - - [ ] Run `Package-Toolkit -ZipPBI` script. + > _Queries are already trimmed to what each report needs. There is nothing to remove by hand._ + - [ ] Run `Package-PowerBI.ps1` to validate the saved files and create PowerBI-demo.zip. + - [ ] Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. - [ ] Check the docs for broken links: - Create a personal fork of the main repo. - If you already have one, update it to the latest. diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 767edf7ba..6cb44efdf 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -129,8 +129,11 @@ _Released June 2026_ - **Changed** - Updated report labels and documentation to use current FinOps Framework capability names, including Usage optimization and Governance, policy, and risk ([#2170](https://github.com/microsoft/finops-toolkit/pull/2170)). - **Fixed** + - Fixed report templates opening with a preconfigured demo data source. Storage and cluster settings are now empty when you open a template. - Fixed Power BI storage report refresh errors caused by ISO 8601 duration `x_SkuTerm` values (like `P3Y`) and empty strings in cost exports ([#2174](https://github.com/microsoft/finops-toolkit/issues/2174)). - Paginate Azure Resource Graph queries by subscription to mitigate [payload size limit](help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) errors in the [Governance](power-bi/governance.md) and [Workload optimization](power-bi/workload-optimization.md) reports ([#1768](https://github.com/microsoft/finops-toolkit/issues/1768)). Queries still surface a payload size error (rather than silently returning truncated results) if a single batch of subscriptions exceeds the limit; see [Reduce the batch size](help/errors.md#option-1-reduce-the-batch-size) for the mitigation. As part of this change, the `AdvisorRecommendations` and `AdvisorReservationRecommendations` tables now return complete (non-truncated) results per batch, matching every other batched table; previously these two tables silently dropped rows beyond the payload limit instead of erroring. + - Fixed the [Governance](power-bi/governance.md) report template missing the policy definitions query. + - Fixed report templates showing tables in the model diagram that the report doesn't use. ### [FinOps workbooks](workbooks/finops-workbooks-overview.md) v15 diff --git a/docs-wiki/Build-and-test.md b/docs-wiki/Build-and-test.md index 8410afcd9..9154d1802 100644 --- a/docs-wiki/Build-and-test.md +++ b/docs-wiki/Build-and-test.md @@ -132,16 +132,19 @@ The Build-PowerShell script calls an Invoke-Build task internally. ### Building Power BI reports -Power BI reports cannot be built automatically. Generating PBIT and PBIX files must be done manually: - -1. Open the desired Power BI report. -2. Save the report as a `.pbix` file using the same name in the `releases` folder. -3. Remove unnecessary queries. -4. Remove setup instructions for unnecessary parameters. -5. Remove unnecessary parameters. -6. Save the report again. -7. Copy the first paragraph description from the main page. -8. Save the report as a `.pbit` file using the copied description and add "To learn more, see https://aka.ms/ftk/". +Run [Build-PowerBI](../src/scripts/README.md#-build-powerbi) to generate the Power BI release artifacts: + +```powershell +cd /src/scripts +./Build-PowerBI +``` + +This generates two things from a single prune, so the template and the demo report can't drift: + +- PBIT templates in `release/pbit`, zipped into `PowerBI-kql.zip` and `PowerBI-storage.zip`. +- PBIP projects in `release/pbix`, each containing only the tables, relationships, and queries that report needs. + +Power BI Desktop has to load and save the demo PBIX files, so that step is still manual. Use [Package-PowerBI](../src/scripts/README.md#-package-powerbi), which reports what's left to do and validates the saved files. ### Building documentation diff --git a/docs-wiki/Release-process.md b/docs-wiki/Release-process.md index 55deb704f..404c88f61 100644 --- a/docs-wiki/Release-process.md +++ b/docs-wiki/Release-process.md @@ -121,21 +121,25 @@ Once the above requirements have been met, the feature branch can be merged into > _The documentation site may take 5 minutes to update after the merge is committed. If not updated, look at [GitHub actions](../actions/workflows/pages/pages-build-deployment) to see if there are any failures._ - 6. Run `Package-Toolkit -Build -PowerBI` script. - - For each Power BI report: - 1. Save the file as a PBIX file to the release folder. - 2. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠️ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - 3. Update the version on the **Get started** tab. - 4. For the Cost summary and Data ingestion reports, remove the following from the Transform data (query editor): - 1. Delete both **Recommendations\*** queries. - 2. Delete the **InstanceSizeFlexibility** query. - 3. Open the **▶️ START HERE** query in the advanced editor and remove connector settings and generated rows in the table from the separator line to the end. - 5. For the Cost summary and Rate optimization reports, remove the following from the Transform data (query editor): - 1. Delete all **Hub\*** queries. - 6. Save PBIX again in the release folder. - > ⚠️ _**DO NOT** save the above changes back to the Power BI project files!_ - 7. Copy the first paragraph from the **Get started** page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/{report-name}" as a separate paragraph in the description. + 6. Package the Power BI files: + + ```powershell + cd /src/scripts + ./Package-PowerBI + ``` + + > _This command is resumable. Run it, do what it asks, then run it again. It reports what's done, what's left, and checks the saved PBIX files for missed steps._ + + 1. Run `./Package-PowerBI -Open` to open the projects that still need to be saved. + 2. For each Power BI project that opens: + 1. Refresh the report so demo data is loaded. + 2. Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. + 3. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. + > ⚠️ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ + 4. Verify all pages, switch to the **Get started** page, and save again. + > _Queries are already trimmed to what each report needs. There is nothing to remove by hand, and the version is already stamped._ + 3. Run `./Package-PowerBI` again to validate the saved files and create PowerBI-demo.zip. + 4. Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. 7. Tag and publish a [new release](../releases/new): 1. Create a tag on publish using the "vX.X" format. 2. Set the **Target** to `main`. diff --git a/src/scripts/Build-PowerBI.ps1 b/src/scripts/Build-PowerBI.ps1 index f65f33fa3..226b7cf3a 100644 --- a/src/scripts/Build-PowerBI.ps1 +++ b/src/scripts/Build-PowerBI.ps1 @@ -3,7 +3,19 @@ <# .SYNOPSIS - Creates PBIT files for each Power BI project. + Builds Power BI release artifacts: PBIT templates and pruned PBIP projects. + + .DESCRIPTION + Generates one PBIT template per report and, unless -NoPbip is specified, one self-contained + PBIP project per report under the release/pbix folder. + + Each generated PBIP contains only the tables, relationships, and queries that its report + needs. That removes the manual "remove unused queries" step when saving demo PBIX files and + guarantees the PBIT and the PBIX are built from the same model, because both are rendered + from a single prune. + + Tables that no report uses are dropped before the model is loaded, so a broken table that + isn't shipped can never block a release. .PARAMETER Name Optional. Name of the report to build. Wildcards supported. Default = * (all). @@ -14,15 +26,23 @@ .PARAMETER Storage Optional. Indicates if the storage reports should be generated. Default = false (will build all if no types are selected). + .PARAMETER NoPbip + Optional. Skips generating pruned PBIP projects and only builds PBIT files. Default = false. + .EXAMPLE ./Build-PowerBI - Generates all PBIT files. + Generates all PBIT files and pruned PBIP projects. .EXAMPLE ./Build-PowerBI CostSummary -Storage - Generates the Cost summary storage PBIT file. + Generates the Cost summary storage PBIT file and PBIP project. + + .EXAMPLE + ./Build-PowerBI -NoPbip + + Generates PBIT files only. .LINK https://github.com/microsoft/finops-toolkit/blob/dev/src/scripts/README.md#-build-powerbi @@ -36,28 +56,394 @@ param( $KQL, [switch] - $Storage + $Storage, + + [switch] + $NoPbip ) +$ErrorActionPreference = 'Stop' + $srcDir = "$PSScriptRoot/../power-bi" $relDir = "$PSScriptRoot/../../release" -$pbipDir = "$relDir/pbip" $pbitDir = "$relDir/pbit" +$pbixDir = "$relDir/pbix" -$version = & "$PSScriptRoot/Get-Version" +$version = & "$PSScriptRoot/Get-Version.ps1" +$buildDate = Get-Date -Format 'yyyy-MM-dd' if (-not $KQL -and -not $Storage) { $KQL = $Storage = $true } +#region Report metadata + +# Tables and queries to keep in each report. Prefix a name with [kql] or [storage] to keep it +# for that report type only. Everything not listed here is removed from the PBIT and the PBIP. +$reportMetadata = @{ + CostSummary = @{ + Intro = "The Cost summary report provides several summaries of your effective (amortized) and billed costs based on the FinOps Open Cost and Usage Specification (FOCUS). Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit. Effective costs will not match your invoice." + Tables = @("Costs", "Prices", "PricingUnits") + Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } + Invoicing = @{ + Intro = "The Invoicing and chargeback report provides several summaries of your billed cost to facilitate invoice reconciliation for Microsoft Customer Agreement (MCA) and Enterprise Agreement (EA) accounts or to perform chargeback using effective (amortized) costs." + Tables = @("Costs", "Prices", "PricingUnits") + Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } + DataIngestion = @{ + Intro = "The Data ingestion report provides details about the data you've ingested into your FinOps hub storage account." + Tables = @("Costs", "HubScopes", "HubSettings", "Prices", "PricingUnits", "StorageData", "StorageErrors") + Expressions = @("▶️ START HERE", "Cluster URL", "Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } + Governance = @{ + Intro = "The Governance, policy, and risk report summarizes your Microsoft Cloud governance posture. It offers the standard metrics aligned with the Cloud Adoption Framework to facilitate identifying issues, applying recommendations, and resolving compliance gaps." + Tables = @("AdvisorRecommendations", "Compliance calculation", "Costs", "Disks", "ManagementGroups", "NetworkInterfaces", "NetworkSecurityGroups", "PolicyAssignments", "PolicyStates", "Prices", "PricingUnits", "PublicIPAddresses", "Regions", "Resources", "ResourceTypes", "SqlDatabases", "Subscriptions", "VirtualMachines") + Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "PolicyDefinitions", "Remove Duplicate Resource IDs", "Deprecated: Perform Extra Query Optimizations", "ftk_ARGBatchSize", "ftk_DemoFilter", "ftk_QueryARG", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } + RateOptimization = @{ + Intro = "The Rate optimization report provides insights into any rate optimization opportunities, like reservations, savings plans, and Azure Hybrid Benefit. This report uses effective cost, which amortizes and breaks reservation and savings plan purchases down and allocates costs out to the resources that received the benefit. Effective cost will not match your invoice." + Tables = @("Costs", "InstanceSizeFlexibility", "Prices", "PricingUnits", "ReservationRecommendations") + Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } + WorkloadOptimization = @{ + Intro = "The Usage optimization report provides insights into resource utilization and efficiency opportunities based on historical usage patterns. Use this report to determine if resources can be scaled down or even shut down during off-peak hours to minimize wasteful usage and spending. Also consider cheaper alternatives when available and ensure all workloads have some direct or indirect link to business value to avoid unnecessary usage and costs that don't contribute to the mission." + Tables = @("AdvisorRecommendations", "Costs", "Disks", "Prices", "PricingUnits", "Resources", "Subscriptions") + Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Remove Duplicate Resource IDs", "Deprecated: Perform Extra Query Optimizations", "ftk_ARGBatchSize", "ftk_DemoFilter", "ftk_QueryARG", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + } +} + +#endregion Report metadata + +#region Helpers + +$script:Utf8NoBom = New-Object System.Text.UTF8Encoding($false) + +function Write-TextFile($Path, [string[]] $Lines) +{ + [System.IO.File]::WriteAllText($Path, (($Lines -join "`n") + "`n"), $script:Utf8NoBom) +} + +function Write-UTF16LE($File, $Content, $Json) +{ + Write-Verbose " Writing UTF-16LE file: $(Split-Path $File -Leaf)..." + if ($Json) { $Content = [PSCustomObject]$Json | ConvertTo-Json -Depth 5 -Compress } + [System.IO.File]::WriteAllBytes($File, [System.Text.Encoding]::Unicode.GetBytes($Content)) +} + +<# + .SYNOPSIS + Reads the name from a TMDL declaration (for example, "column 'Foo Bar' = 1" returns "Foo Bar"). +#> +function ConvertFrom-TmdlDeclaration([string] $Text) +{ + $value = $Text.Trim() + if ($value.StartsWith("'")) + { + for ($i = 1; $i -lt $value.Length; $i++) + { + if ($value[$i] -ne "'") { continue } + if ($i + 1 -lt $value.Length -and $value[$i + 1] -eq "'") { $i++; continue } + return $value.Substring(1, $i - 1).Replace("''", "'") + } + return $null + } + + $stop = $value.IndexOfAny([char[]]@(' ', "`t", '=')) + if ($stop -lt 0) { return $value } + return $value.Substring(0, $stop) +} + +<# + .SYNOPSIS + Reads the table name from a TMDL column reference (for example, "'My Table'.Col" returns "My Table"). +#> +function Get-TmdlReferenceTable([string] $Reference) +{ + $value = $Reference.Trim() + if ($value.StartsWith("'")) + { + for ($i = 1; $i -lt $value.Length; $i++) + { + if ($value[$i] -ne "'") { continue } + if ($i + 1 -lt $value.Length -and $value[$i + 1] -eq "'") { $i++; continue } + return $value.Substring(1, $i - 1).Replace("''", "'") + } + return $null + } + + $dot = $value.IndexOf('.') + if ($dot -lt 0) { return $value } + return $value.Substring(0, $dot) +} + +<# + .SYNOPSIS + Resolves an allowlist, keeping entries that apply to the specified report type. +#> +function Resolve-Allowlist([string[]] $Entries, [string] $ReportType) +{ + $resolved = New-Object System.Collections.Generic.List[string] + foreach ($entry in $Entries) + { + if ($entry -match '^\[(?[^\]]+)\](?.+)$') + { + if ($Matches.type -eq $ReportType) { $resolved.Add($Matches.name) } + } + else + { + $resolved.Add($entry) + } + } + return , $resolved.ToArray() +} + +function ConvertTo-NameSet([string[]] $Names) +{ + return New-Object System.Collections.Generic.HashSet[string]([string[]]$Names, [System.StringComparer]::OrdinalIgnoreCase) +} + +<# + .SYNOPSIS + Finds names that only differ by case, which Power BI rejects when loading the model. + + .DESCRIPTION + Tabular models treat names as case-insensitive, so two columns named "region" and "Region" + cannot coexist. Power BI Desktop and the TMDL text format both allow the file to be saved + that way, and the failure only shows up when the model is loaded. Detecting it here turns a + confusing load failure into a specific message that names the file, table, and lines. +#> +function Get-TmdlNameConflict([string] $DefinitionDir) +{ + $conflicts = New-Object System.Collections.Generic.List[object] + + function Add-Conflict($List, $Table, $File, $Names) + { + $Names.GetEnumerator() ` + | Where-Object { $_.Value.Count -gt 1 } ` + | ForEach-Object { + $scope = if ($Table) { "table '$Table'" } else { 'the query list' } + $conflicting = ($_.Value | ForEach-Object { $_.Name }) -join ', ' + $lines = ($_.Value | ForEach-Object { $_.Line }) -join ', ' + $List.Add([PSCustomObject]@{ + Table = $Table + File = $File + Message = "$File`: $scope declares names that only differ by case ($conflicting) on lines $lines. Power BI treats names as case-insensitive, so one of them must be renamed or removed." + }) + } + } + + Get-ChildItem "$DefinitionDir/tables" -Filter '*.tmdl' -ErrorAction SilentlyContinue ` + | ForEach-Object { + $file = $_ + $lines = [System.IO.File]::ReadAllLines($file.FullName) + $tableName = $file.BaseName + $members = @{} + + for ($i = 0; $i -lt $lines.Count; $i++) + { + $line = $lines[$i] + if ($line -match '^table\s+(?.+)$') + { + $tableName = ConvertFrom-TmdlDeclaration $Matches.rest + } + elseif ($line -match '^\t(?column|measure)\s+(?.+)$') + { + $memberName = ConvertFrom-TmdlDeclaration $Matches.rest + if (-not $memberName) { continue } + $key = $memberName.ToLowerInvariant() + if (-not $members.ContainsKey($key)) { $members[$key] = New-Object System.Collections.Generic.List[object] } + $members[$key].Add([PSCustomObject]@{ Name = $memberName; Line = $i + 1 }) + } + } + + Add-Conflict $conflicts $tableName $file.Name $members + } + + $expressionFile = "$DefinitionDir/expressions.tmdl" + if (Test-Path $expressionFile) + { + $lines = [System.IO.File]::ReadAllLines($expressionFile) + $expressions = @{} + for ($i = 0; $i -lt $lines.Count; $i++) + { + if ($lines[$i] -notmatch '^expression\s+(?.+)$') { continue } + $expressionName = ConvertFrom-TmdlDeclaration $Matches.rest + if (-not $expressionName) { continue } + $key = $expressionName.ToLowerInvariant() + if (-not $expressions.ContainsKey($key)) { $expressions[$key] = New-Object System.Collections.Generic.List[object] } + $expressions[$key].Add([PSCustomObject]@{ Name = $expressionName; Line = $i + 1 }) + } + Add-Conflict $conflicts $null 'expressions.tmdl' $expressions + } + + return , $conflicts.ToArray() +} + +<# + .SYNOPSIS + Copies a semantic model, keeping only the specified tables. + + .DESCRIPTION + Tables are removed from disk before the model is loaded so that tables no report ships + cannot block the build. Relationships that point at a removed table and the matching + "ref table" entries in model.tmdl are removed at the same time to keep the model loadable. +#> +function Export-PrunedDataset([string] $SourceDir, [string] $TargetDir, [string[]] $KeepTables) +{ + $keep = ConvertTo-NameSet $KeepTables + + Remove-Item $TargetDir -Recurse -Force -ErrorAction SilentlyContinue + & "$PSScriptRoot/New-Directory.ps1" (Split-Path $TargetDir -Parent) + Copy-Item $SourceDir $TargetDir -Recurse -Force + + $definitionDir = "$TargetDir/definition" + $dropped = New-Object System.Collections.Generic.List[string] + $kept = New-Object System.Collections.Generic.List[string] + + # Remove table files that aren't needed + Get-ChildItem "$definitionDir/tables" -Filter '*.tmdl' ` + | ForEach-Object { + $file = $_ + $tableName = $file.BaseName + foreach ($line in [System.IO.File]::ReadAllLines($file.FullName)) + { + if ($line -match '^table\s+(?.+)$') { $tableName = ConvertFrom-TmdlDeclaration $Matches.rest; break } + } + + if ($keep.Contains($tableName)) + { + $kept.Add($tableName) + } + else + { + Write-Verbose " Removing table: $tableName" + $dropped.Add($tableName) + Remove-Item $file.FullName -Force + } + } + + $droppedSet = ConvertTo-NameSet $dropped + + # Remove "ref table" entries for removed tables + $modelFile = "$definitionDir/model.tmdl" + $modelLines = [System.IO.File]::ReadAllLines($modelFile) + $keptModelLines = $modelLines | Where-Object { + if ($_ -notmatch '^ref table\s+(?.+)$') { return $true } + return -not $droppedSet.Contains((ConvertFrom-TmdlDeclaration $Matches.rest)) + } + Write-TextFile $modelFile $keptModelLines + + # Remove relationships that point at a removed table + $removedRelationships = 0 + $relationshipFile = "$definitionDir/relationships.tmdl" + if (Test-Path $relationshipFile) + { + $lines = [System.IO.File]::ReadAllLines($relationshipFile) + $starts = @(0..($lines.Count - 1) | Where-Object { $lines[$_] -match '^relationship\s' }) + + if ($starts.Count -gt 0) + { + $output = New-Object System.Collections.Generic.List[string] + for ($i = 0; $i -lt $starts[0]; $i++) { $output.Add($lines[$i]) } + + for ($b = 0; $b -lt $starts.Count; $b++) + { + $start = $starts[$b] + $end = if ($b + 1 -lt $starts.Count) { $starts[$b + 1] - 1 } else { $lines.Count - 1 } + + $block = New-Object System.Collections.Generic.List[string] + $referencesDropped = $false + for ($i = $start; $i -le $end; $i++) + { + $block.Add($lines[$i]) + if ($lines[$i] -match '^\s*(from|to)Column:\s*(?.+)$') + { + if ($droppedSet.Contains((Get-TmdlReferenceTable $Matches.ref))) { $referencesDropped = $true } + } + } + + if ($referencesDropped) + { + $removedRelationships++ + continue + } + + while ($block.Count -gt 0 -and [string]::IsNullOrWhiteSpace($block[$block.Count - 1])) { $block.RemoveAt($block.Count - 1) } + $block | ForEach-Object { $output.Add($_) } + $output.Add('') + } + + Write-TextFile $relationshipFile $output + } + } + + # Drop model diagram nodes for removed tables so the diagram matches the model + $diagramFile = "$TargetDir/diagramLayout.json" + if (Test-Path $diagramFile) + { + $diagram = Get-Content $diagramFile -Raw | ConvertFrom-Json -Depth 100 + foreach ($layout in $diagram.diagrams) + { + $layout.nodes = @($layout.nodes | Where-Object { $keep.Contains($_.nodeIndex) }) + } + $diagram | ConvertTo-Json -Depth 100 | Set-Content $diagramFile -NoNewline + } + + return [PSCustomObject]@{ + Kept = $kept.ToArray() + Dropped = $dropped.ToArray() + RemovedRelationships = $removedRelationships + } +} + +<# + .SYNOPSIS + Stamps the version into a report and confirms the Get started page is the active page. +#> +function Get-ReportLayout([string] $ReportDir, [string] $ReportLabel) +{ + $json = (Get-Content "$ReportDir/report.json" -Raw) ` + -replace '\$\$ftkver\$\$', $version ` + -replace '\$\$build-date\$\$', $buildDate + + $report = $json | ConvertFrom-Json -Depth 100 + + # Pages are stored in an arbitrary order and displayed by ordinal, so the active page index + # is an index into the display order, not into the sections array. + $displayOrder = @($report.sections | Sort-Object @{ Expression = { if ($null -eq $_.ordinal) { 0 } else { $_.ordinal } } }) + $expected = [array]::FindIndex($displayOrder, [Predicate[object]] { param($section) $section.displayName -eq 'Get started' }) + + if ($expected -lt 0) + { + Write-Warning "$ReportLabel has no 'Get started' page. Leaving the active page unchanged." + return $json + } + + $config = $report.config | ConvertFrom-Json -Depth 100 + if ($config.activeSectionIndex -eq $expected) { return $json } + + Write-Warning "$ReportLabel opens on '$($displayOrder[$config.activeSectionIndex].displayName)'. Resetting to 'Get started'." + $config.activeSectionIndex = $expected + $report.config = $config | ConvertTo-Json -Depth 100 -Compress + return ($report | ConvertTo-Json -Depth 100) +} + +#endregion Helpers + +#region Setup + +& "$PSScriptRoot/New-Directory.ps1" $relDir +& "$PSScriptRoot/New-Directory.ps1" $pbitDir + # Cleanup Write-Verbose "Removing existing ZIP files..." if ($KQL) -{ - Remove-Item "$pbitDir/../PowerBI-kql.zip" -Force -ErrorAction SilentlyContinue +{ + Remove-Item "$relDir/PowerBI-kql.zip" -Force -ErrorAction SilentlyContinue Remove-Item "$pbitDir/*.kql.pbit" -Force -ErrorAction SilentlyContinue } if ($Storage) -{ - Remove-Item "$pbitDir/../PowerBI-storage.zip" -Force -ErrorAction SilentlyContinue +{ + Remove-Item "$relDir/PowerBI-storage.zip" -Force -ErrorAction SilentlyContinue Remove-Item "$pbitDir/*.storage.pbit" -Force -ErrorAction SilentlyContinue } @@ -67,36 +453,17 @@ if ($KQL) { $types += "*$Name*.kql.pbip" } # cSpell:ignore PBIP if ($Storage) { $types += "*$Name*.storage.pbip" } # Get reports -$reports = Get-ChildItem $srcDir -Recurse -Include $types -Write-Host "Building $($reports.Count) Power BI report template$(if ($reports.Count -ne 1) { 's' })..." - -function Write-UTF16LE($File, $Content, $Json) +$reports = @(Get-ChildItem $srcDir -Recurse -Include $types | Sort-Object FullName) +if ($reports.Count -eq 0) { - Write-Verbose " Writing UTF-16LE file: $($File.Name)..." - if ($Json) { $Content = [PSCustomObject]$Json | ConvertTo-Json -Depth 5 -Compress } - [System.IO.File]::WriteAllBytes($File, [System.Text.Encoding]::Unicode.GetBytes($Content)) + throw "No Power BI projects matched '$Name'. Confirm the report name." } +Write-Host "Building $($reports.Count) Power BI report$(if ($reports.Count -ne 1) { 's' })..." # Setup dependencies if (-not (Get-Package Microsoft.AnalysisServices -ErrorAction SilentlyContinue)) { - # $adminCheck = [Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent() - # $isAdmin = $adminCheck.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) - - # Write-Host "The Microsoft.AnalysisServices package is required to build the data model." - # if ($isAdmin) - # { - # Write-Host " This window cannot run admin commands." - # Write-Host " ⚠️ Please run this command in an admin elevated prompt: Install-Package Microsoft.AnalysisServices -ProviderName NuGet" - # return - # } - # else - # { - # Write-Host " Installing the the Analysis Services package..." - # Install-Package Microsoft.AnalysisServices -Force - # } - - Write-Verbose " Installing the the Analysis Services package..." + Write-Verbose " Installing the Analysis Services package..." Write-Verbose " PRO TIP: Install via an admin prompt to speed up future runs!" Install-Package -Name Microsoft.AnalysisServices -ProviderName NuGet -Scope CurrentUser -Force } @@ -105,104 +472,174 @@ $dllPath = "$((Get-Item (Get-Package Microsoft.AnalysisServices).Source).Directo Write-Verbose " Adding type from $dllPath" Add-Type -Path $dllPath -# Loop thru reports -$reports | ForEach-Object { - $inputFile = $_ +#endregion Setup + +#region Build + +$conflictCache = @{} +$manifest = New-Object System.Collections.Generic.List[object] +$allowlistEntries = New-Object System.Collections.Generic.HashSet[string] +$allowlistMatches = New-Object System.Collections.Generic.HashSet[string] + +foreach ($inputFile in $reports) +{ $reportName = $inputFile.Name.Split('.')[0] + $reportType = $inputFile.Name.Split('.')[1] # Extract "kql" or "storage" from filename + $baseName = "$reportName.$reportType" $folder = $inputFile.DirectoryName $reportDir = "$folder/$reportName.Report" $datasetDir = "$folder/Shared.Dataset" - $reportType = $inputFile.Name.Split('.')[1] # Extract "kql" or "storage" from filename - Write-Verbose "Processing $($inputFile.Name)..." - $metadata = @{ - CostSummary = @{ - Intro = "The Cost summary report provides several summaries of your effective (amortized) and billed costs based on the FinOps Open Cost and Usage Specification (FOCUS). Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit. Effective costs will not match your invoice." - Tables = @("Costs", "Prices", "PricingUnits") - Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") - } - Invoicing = @{ - Intro = "The Invoicing and chargeback report provides several summaries of your billed cost to facilitate invoice reconciliation for Microsoft Customer Agreement (MCA) and Enterprise Agreement (EA) accounts or to perform chargeback using effective (amortized) costs." - Tables = @("Costs", "Prices", "PricingUnits") - Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") - } - DataIngestion = @{ - Intro = "The Data ingestion report provides details about the data you've ingested into your FinOps hub storage account." - Tables = @("Costs", "HubScopes", "HubSettings", "Prices", "PricingUnits", "StorageData", "StorageErrors") - Expressions = @("▶️ START HERE", "Cluster URL", "Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") - } - Governance = @{ - Intro = "The Governance, policy, and risk report summarizes your Microsoft Cloud governance posture. It offers the standard metrics aligned with the Cloud Adoption Framework to facilitate identifying issues, applying recommendations, and resolving compliance gaps." - Tables = @("AdvisorRecommendations", "Compliance calculation", "Costs", "Disks", "ManagementGroups", "NetworkInterfaces", "NetworkSecurityGroups", "PolicyAssignments", "PolicyDefinitions", "PolicyStates", "Prices", "PricingUnits", "PublicIPAddresses", "Regions", "Resources", "ResourceTypes", "SqlDatabases", "Subscriptions", "VirtualMachines") - Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Remove Duplicate Resource IDs", "Deprecated: Perform Extra Query Optimizations", "ftk_ARGBatchSize", "ftk_DemoFilter", "ftk_QueryARG", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") - } - RateOptimization = @{ - Intro = "The Rate optimization report provides insights into any rate optimization opportunities, like reservations, savings plans, and Azure Hybrid Benefit. This report uses effective cost, which amortizes and breaks reservation and savings plan purchases down and allocates costs out to the resources that received the benefit. Effective cost will not match your invoice." - Tables = @("Costs", "InstanceSizeFlexibility", "Prices", "PricingUnits", "ReservationRecommendations") - Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Deprecated: Perform Extra Query Optimizations", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + Write-Host " $baseName..." + + $metadata = $reportMetadata[$reportName] + if (-not $metadata) { throw "No build metadata is defined for the '$reportName' report. Add it to the `$reportMetadata table in Build-PowerBI.ps1." } + + $keepTables = Resolve-Allowlist $metadata.Tables $reportType + $keepExpressions = Resolve-Allowlist $metadata.Expressions $reportType + + # Fail on name conflicts this report ships; report the rest once per model + if (-not $conflictCache.ContainsKey($datasetDir)) + { + $conflictCache[$datasetDir] = Get-TmdlNameConflict "$datasetDir/definition" + $conflictCache[$datasetDir] | ForEach-Object { Write-Warning "$($_.Message) It was left out of this build." } + } + foreach ($conflict in $conflictCache[$datasetDir]) + { + if ($conflict.Table -and -not ($keepTables -contains $conflict.Table)) { continue } + throw "$($conflict.Message) The $baseName report needs it, so the build cannot continue." + } + + # Prune the model on disk, then load it + $stagedDataset = "$pbixDir/$baseName.Dataset" + $pruned = Export-PrunedDataset $datasetDir $stagedDataset $keepTables + Write-Verbose " Kept $($pruned.Kept.Count) tables, dropped $($pruned.Dropped.Count), removed $($pruned.RemovedRelationships) relationships" + + $db = [Microsoft.AnalysisServices.Tabular.TmdlSerializer]::DeserializeDatabaseFromFolder("$stagedDataset/definition") # cSpell:ignore TMDL + + # Remove queries this report doesn't use + $keepExpressionSet = ConvertTo-NameSet $keepExpressions + @($db.Model.Expressions) ` + | Where-Object { -not $keepExpressionSet.Contains($_.Name) } ` + | ForEach-Object { + Write-Verbose " Removing query: $($_.Name)" + $db.Model.Expressions.Remove($_) | Out-Null + } + + # Track allowlist entries that match nothing. These are only reported once every report type + # has been built, because most entries only exist in one of the two models by design. + $modelNames = ConvertTo-NameSet (@($db.Model.Tables | ForEach-Object { $_.Name }) + @($db.Model.Expressions | ForEach-Object { $_.Name })) + foreach ($entry in @($keepTables + $keepExpressions)) + { + $null = $allowlistEntries.Add("$reportName|$entry") + if ($modelNames.Contains($entry)) { $null = $allowlistMatches.Add("$reportName|$entry") } + } + + # Rebuild the query pane order from what survived + $queryOrder = $db.Model.Annotations | Where-Object { $_.Name -eq 'PBI_QueryOrder' } | Select-Object -First 1 + if ($queryOrder) + { + $surviving = @($db.Model.Expressions | ForEach-Object { $_.Name }) + @($db.Model.Tables | ForEach-Object { $_.Name }) + $survivingSet = ConvertTo-NameSet $surviving + + $previous = @() + try { $previous = @($queryOrder.Value | ConvertFrom-Json) } catch { Write-Verbose " Could not read the existing query order; rebuilding it." } + + $ordered = New-Object System.Collections.Generic.List[string] + $previous | Where-Object { $survivingSet.Contains($_) } | ForEach-Object { $ordered.Add($_) } + $surviving | Where-Object { $ordered -notcontains $_ } | ForEach-Object { $ordered.Add($_) } + + $queryOrder.Value = $ordered | ConvertTo-Json -Depth 1 -Compress -AsArray + } + + #region PBIP project + + if (-not $NoPbip) + { + # Re-serialize the pruned model so the PBIP matches what the PBIT ships + Remove-Item "$stagedDataset/definition" -Recurse -Force + [Microsoft.AnalysisServices.Tabular.TmdlSerializer]::SerializeDatabaseToFolder($db, "$stagedDataset/definition") + + $platformFile = "$stagedDataset/.platform" + if (Test-Path $platformFile) + { + $platform = Get-Content $platformFile -Raw | ConvertFrom-Json + $platform.metadata.displayName = $baseName + $platform.config.logicalId = [guid]::NewGuid().ToString() + $platform | ConvertTo-Json -Depth 10 | Set-Content $platformFile -NoNewline } - WorkloadOptimization = @{ - Intro = "The Usage optimization report provides insights into resource utilization and efficiency opportunities based on historical usage patterns. Use this report to determine if resources can be scaled down or even shut down during off-peak hours to minimize wasteful usage and spending. Also consider cheaper alternatives when available and ensure all workloads have some direct or indirect link to business value to avoid unnecessary usage and costs that don't contribute to the mission." - Tables = @("AdvisorRecommendations", "Costs", "Disks", "Prices", "PricingUnits", "Resources", "Subscriptions") - Expressions = @("▶️ START HERE", "Cluster URL", "[storage]Storage URL", "Default Granularity", "Number of Months", "RangeStart", "RangeEnd", "Experimental: Add Missing Prices", "Remove Duplicate Resource IDs", "Deprecated: Perform Extra Query Optimizations", "ftk_ARGBatchSize", "ftk_DemoFilter", "ftk_QueryARG", "ftk_DatetimeToJulianDate", "ftk_ImpalaToJulianDate", "ftk_Metadata", "ftk_ParseResourceId", "ftk_ParseResourceName", "ftk_ParseResourceType", "ftk_Storage") + + # Report + $stagedReport = "$pbixDir/$baseName.Report" + Remove-Item $stagedReport -Recurse -Force -ErrorAction SilentlyContinue + Copy-Item $reportDir $stagedReport -Recurse -Force + + [System.IO.File]::WriteAllText("$stagedReport/report.json", (Get-ReportLayout $reportDir $baseName), $script:Utf8NoBom) + + @{ + '$schema' = 'https://developer.microsoft.com/json-schemas/fabric/item/report/definitionProperties/1.0.0/schema.json' + version = '4.0' + datasetReference = @{ byPath = @{ path = "../$baseName.Dataset" } } + } | ConvertTo-Json -Depth 10 | Set-Content "$stagedReport/definition.pbir" -NoNewline + + $reportPlatformFile = "$stagedReport/.platform" + if (Test-Path $reportPlatformFile) + { + $reportPlatform = Get-Content $reportPlatformFile -Raw | ConvertFrom-Json + $reportPlatform.metadata.displayName = $baseName + $reportPlatform.config.logicalId = [guid]::NewGuid().ToString() + $reportPlatform | ConvertTo-Json -Depth 10 | Set-Content $reportPlatformFile -NoNewline } - }[$reportName] - # Create folder structure - $targetFile = "$pbitDir/$($inputFile.Name.Replace('.pbip', ''))" + # Project file + @{ + '$schema' = 'https://developer.microsoft.com/json-schemas/fabric/pbip/pbipProperties/1.0.0/schema.json' + version = '1.0' + artifacts = @(@{ report = @{ path = "$baseName.Report" } }) + settings = @{ enableAutoRecovery = $true } + } | ConvertTo-Json -Depth 10 | Set-Content "$pbixDir/$baseName.pbip" -NoNewline + } + + #endregion PBIP project + + #region PBIT template + + $targetFile = "$pbitDir/$baseName" Remove-Item $targetFile -Recurse -Force -ErrorAction SilentlyContinue & "$PSScriptRoot/New-Directory.ps1" $targetFile & "$PSScriptRoot/New-Directory.ps1" "$targetFile/Report" # DataModelSchema - $model = [Microsoft.AnalysisServices.Tabular.TmdlSerializer]::DeserializeDatabaseFromFolder("$datasetDir/definition") # cSpell:ignore TMDL - $modelJson = [Microsoft.AnalysisServices.Tabular.JsonSerializer]::SerializeDatabase($model) | ConvertFrom-Json -Depth 100 -AsHashtable + $modelJson = [Microsoft.AnalysisServices.Tabular.JsonSerializer]::SerializeDatabase($db) | ConvertFrom-Json -Depth 100 -AsHashtable $modelJson.name = [guid]::NewGuid() + + # Templates ship without a data source so customers supply their own. The PBIP keeps the + # demo values so demo PBIX files can still refresh. $modelJson.model.expressions ` | ForEach-Object { $exp = $_ if ($exp.name.EndsWith(' URL')) { - $exp.expression = $exp.expression -replace '^\\"[^\\"]+\\" meta ', 'null meta ' + $exp.expression = $exp.expression -replace '^"[^"]*" meta ', 'null meta ' } if ($exp.name -eq 'ftk_DemoFilter') { $exp.expression = '() => "" // To filter out subscriptions, replace with: "| where subscriptionId in ('''', '''')"' } } - $modelJson.model.tables = $modelJson.model.tables | Where-Object { $metadata.Tables -contains $_.name -or $metadata.Tables -contains "[$reportType]$($_.name)" } - $modelJson.model.relationships ` - | Where-Object { -not (($metadata.Tables -contains $_.fromTable -or $metadata.Tables -contains "[$reportType]$($_.fromTable)") -and ($metadata.Tables -contains $_.toTable -or $metadata.Tables -contains "[$reportType]$($_.toTable)")) } ` - | ForEach-Object { Write-Verbose " Removing relationship: $($_.fromTable) -> $($_.toTable)" } - $modelJson.model.relationships = @($modelJson.model.relationships | Where-Object { ($metadata.Tables -contains $_.fromTable -or $metadata.Tables -contains "[$reportType]$($_.fromTable)") -and ($metadata.Tables -contains $_.toTable -or $metadata.Tables -contains "[$reportType]$($_.toTable)") }) - $modelJson.model.expressions = $modelJson.model.expressions | Where-Object { $metadata.Expressions -contains $_.name -or $metadata.Expressions -contains "[$reportType]$($_.name)" } - $modelJson.model.annotations = $modelJson.model.annotations ` - | ForEach-Object { - $ann = $_ - if ($ann.name -eq 'PBI_QueryOrder') - { - $ann.value = ($metadata.Expressions + $metadata.Tables) | ConvertTo-Json -Depth 1 -Compress | ConvertTo-Json -Depth 1 - } - return $ann - } + Write-UTF16LE -File "$targetFile/DataModelSchema" -Content ($modelJson | ConvertTo-Json -Depth 100) - # DiagramLayout - Write-UTF16LE -File "$targetFile/DiagramLayout" -Json (Get-Content "$datasetDir/diagramLayout.json" -Raw | ConvertFrom-Json -Depth 100) + # DiagramLayout (from the pruned model so it doesn't reference tables that were removed) + Write-UTF16LE -File "$targetFile/DiagramLayout" -Json (Get-Content "$stagedDataset/diagramLayout.json" -Raw | ConvertFrom-Json -Depth 100) # Report/Layout - $reportJson = (Get-Content "$reportDir/report.json" -Raw) ` - -replace '\$\$ftkver\$\$', $version ` - -replace '\$\$build-date\$\$', (Get-Date -Format 'yyyy-MM-dd') - Write-UTF16LE -File "$targetFile/Report/Layout" -Json ($reportJson | ConvertFrom-Json -Depth 100) + Write-UTF16LE -File "$targetFile/Report/Layout" -Json ((Get-ReportLayout $reportDir $baseName) | ConvertFrom-Json -Depth 100) # Report/StaticResources Copy-Item "$reportDir/StaticResources" "$targetFile/Report/StaticResources" -Recurse -Force # Metadata - # TODO: Where does "Version" come from? - # TODO: Add all intro paragraphs - # TODO: Add Load button guidance $desktopVersion = $modelJson.model.annotations | Where-Object { $_.name -eq 'PBIDesktopVersion' } | ForEach-Object { $_.value } Write-Verbose " Desktop version: '$desktopVersion'" Write-UTF16LE -File "$targetFile/Metadata" -Json @{ @@ -210,8 +647,7 @@ $reports | ForEach-Object { AutoCreatedRelationships = @() FileDescription = "$($metadata.Intro)`n`nTo customize queries or data source settings, select the Edit option in the Load button.`n`nLearn more at https://aka.ms/ftk/pbi/$reportName" CreatedFrom = "Cloud" - # TODO: Validate "CreatedFromRelease" - CreatedFromRelease = "20$($desktopVersion -replace '^[^\(]+\(([0-9]{2}\.[0-2][0-9])\)[^\)]+$','$1')" + CreatedFromRelease = "20$($desktopVersion -replace '^[^\(]+\(([0-9]{2}\.[0-2][0-9])\)[^\)]+$', '$1')" } # Settings @@ -225,13 +661,11 @@ $reports | ForEach-Object { QueriesSettings = @{ TypeDetectionEnabled = $editorSettings.typeDetectionEnabled RelationshipImportEnabled = $editorSettings.relationshipImportEnabled - # TODO: Validate "Version" Version = $desktopVersion.Split(' ')[0] } } # Version - # TODO: Where the "Version" file content come from? Write-UTF16LE -File "$targetFile/Version" -Content "1.30" # [Content_Types].xml @@ -252,12 +686,65 @@ $reports | ForEach-Object { | Out-File -LiteralPath "$targetFile/[Content_Types].xml" -Encoding utf8 # Create PBIT file - Compress-Archive -Path "$targetFile/*" -DestinationPath "$targetFile.pbit" - + Compress-Archive -Path "$targetFile/*" -DestinationPath "$targetFile.pbit" -Force Remove-Item $targetFile -Recurse -Force + + #endregion PBIT template + + if (-not $NoPbip) + { + $manifest.Add([PSCustomObject]@{ + name = $reportName + type = $reportType + base = $baseName + pbip = "$baseName.pbip" + pbix = "$baseName.pbix" + pbit = "$baseName.pbit" + tables = @($db.Model.Tables | ForEach-Object { $_.Name } | Sort-Object) + expressions = @($db.Model.Expressions | ForEach-Object { $_.Name } | Sort-Object) + }) + } + else + { + Remove-Item $stagedDataset -Recurse -Force -ErrorAction SilentlyContinue + } } -# Zip files +#endregion Build + +#region Package + $genAllReports = $Name -eq '*' -if ($KQL -and $genAllReports) { Compress-Archive -Path "$pbitDir/*.kql.pbit" -DestinationPath "$pbitDir/../PowerBI-kql.zip" } -if ($Storage -and $genAllReports) { Compress-Archive -Path "$pbitDir/*.storage.pbit" -DestinationPath "$pbitDir/../PowerBI-storage.zip" } +if ($KQL -and $genAllReports) { Compress-Archive -Path "$pbitDir/*.kql.pbit" -DestinationPath "$relDir/PowerBI-kql.zip" -Force } +if ($Storage -and $genAllReports) { Compress-Archive -Path "$pbitDir/*.storage.pbit" -DestinationPath "$relDir/PowerBI-storage.zip" -Force } + +if (-not $NoPbip -and $manifest.Count -gt 0) +{ + @{ + version = $version + built = (Get-Date -Format 'o') + reports = $manifest.ToArray() + } | ConvertTo-Json -Depth 10 | Set-Content "$pbixDir/.manifest.json" +} + +# Allowlist entries that matched nothing in any model built are stale or misspelled +if ($KQL -and $Storage -and $Name -eq '*') +{ + $allowlistEntries ` + | Where-Object { -not $allowlistMatches.Contains($_) } ` + | Sort-Object ` + | ForEach-Object { + $parts = $_.Split('|') + Write-Warning "$($parts[0]) lists '$($parts[1])' but no model has a table or query with that name." + } +} + +Write-Host "✅ $($reports.Count) PBIT template$(if ($reports.Count -ne 1) { 's' })" +if ($genAllReports) +{ + if ($KQL) { Write-Host "✅ PowerBI-kql.zip" } + if ($Storage) { Write-Host "✅ PowerBI-storage.zip" } +} +if (-not $NoPbip) { Write-Host "✅ $($manifest.Count) PBIP project$(if ($manifest.Count -ne 1) { 's' }) in release/pbix" } + +#endregion Package diff --git a/src/scripts/Package-PowerBI.ps1 b/src/scripts/Package-PowerBI.ps1 new file mode 100644 index 000000000..1963ba2e2 --- /dev/null +++ b/src/scripts/Package-PowerBI.ps1 @@ -0,0 +1,365 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + .SYNOPSIS + Packages the three Power BI release files and reports what's left to do. + + .DESCRIPTION + Power BI releases ship three files: PowerBI-kql.zip and PowerBI-storage.zip hold the PBIT + templates, and PowerBI-demo.zip holds the demo PBIX files. + + Everything except saving the PBIX files is automated. Power BI Desktop has to load and save + those, so this command is resumable: run it, save the projects it opens, then run it again. + It works out which steps are already done, does the next one, and validates the result, so a + missed step fails here instead of shipping. + + .PARAMETER Open + Optional. Opens the Power BI projects that still need to be saved as PBIX files. Default = false. + + .PARAMETER Build + Optional. Rebuilds the PBIT templates and PBIP projects even if they already exist. Default = false. + + .PARAMETER Status + Optional. Reports what's done and what's left without changing anything. Default = false. + + .EXAMPLE + ./Package-PowerBI + + Builds whatever is missing and reports the next step. + + .EXAMPLE + ./Package-PowerBI -Open + + Opens the Power BI projects that still need to be saved as PBIX files. + + .EXAMPLE + ./Package-PowerBI -Status + + Reports what's done and what's left. + + .LINK + https://github.com/microsoft/finops-toolkit/blob/dev/src/scripts/README.md#-package-powerbi +#> +param( + [switch] $Open, + + [switch] $Build, + + [switch] $Status +) + +$ErrorActionPreference = 'Stop' + +$relDir = "$PSScriptRoot/../../release" +$pbitDir = "$relDir/pbit" +$pbixDir = "$relDir/pbix" +$manifestPath = "$pbixDir/.manifest.json" + +$version = & "$PSScriptRoot/Get-Version.ps1" + +Add-Type -AssemblyName System.IO.Compression.FileSystem + +#region Helpers + +function Format-Size([long] $Bytes) +{ + if ($Bytes -ge 1GB) { return "{0:N1} GB" -f ($Bytes / 1GB) } + if ($Bytes -ge 1MB) { return "{0:N1} MB" -f ($Bytes / 1MB) } + return "{0:N0} KB" -f ($Bytes / 1KB) +} + +<# + .SYNOPSIS + Reads a single entry from a ZIP-based file (PBIX, PBIT) without extracting it. +#> +function Get-ArchiveEntry([string] $Path, [string] $EntryName) +{ + $archive = [System.IO.Compression.ZipFile]::OpenRead($Path) + try + { + $entry = $archive.Entries | Where-Object { $_.FullName -eq $EntryName } | Select-Object -First 1 + if (-not $entry) { return $null } + + $stream = $entry.Open() + try + { + $buffer = New-Object System.IO.MemoryStream + $stream.CopyTo($buffer) + return $buffer.ToArray() + } + finally { $stream.Dispose() } + } + finally { $archive.Dispose() } +} + +function Get-ArchiveEntryName([string] $Path) +{ + $archive = [System.IO.Compression.ZipFile]::OpenRead($Path) + try { return @($archive.Entries | ForEach-Object { $_.FullName }) } + finally { $archive.Dispose() } +} + +function Get-ArchiveEntrySize([string] $Path, [string] $EntryName) +{ + $archive = [System.IO.Compression.ZipFile]::OpenRead($Path) + try + { + $entry = $archive.Entries | Where-Object { $_.FullName -eq $EntryName } | Select-Object -First 1 + if (-not $entry) { return -1 } + return $entry.Length + } + finally { $archive.Dispose() } +} + +<# + .SYNOPSIS + Parses a JSON part from a PBIX file, which Power BI writes as UTF-16LE. +#> +function ConvertFrom-PbixJson([byte[]] $Bytes) +{ + if ($Bytes.Length -ge 2 -and $Bytes[0] -eq 0xFF -and $Bytes[1] -eq 0xFE) + { + $text = [System.Text.Encoding]::Unicode.GetString($Bytes, 2, $Bytes.Length - 2) + } + elseif ($Bytes.Length -ge 2 -and $Bytes[1] -eq 0) + { + $text = [System.Text.Encoding]::Unicode.GetString($Bytes) + } + else + { + $text = [System.Text.Encoding]::UTF8.GetString($Bytes) + $text = $text.TrimStart([char]0xFEFF) + } + + return $text | ConvertFrom-Json -Depth 100 +} + +<# + .SYNOPSIS + Checks a saved demo PBIX for the mistakes that are easy to make when saving by hand. + + .DESCRIPTION + Catches a report saved without data, saved from the wrong (unpruned) project, saved on the + wrong page, or saved from a stale build with the previous version number. +#> +function Test-DemoPbix([string] $Path, $Report) +{ + $issues = New-Object System.Collections.Generic.List[string] + + try { $entries = Get-ArchiveEntryName $Path } + catch + { + $issues.Add("isn't a readable PBIX file ($($_.Exception.Message)). Save it again from Power BI Desktop.") + return $issues + } + + foreach ($required in @('DataModel', 'Report/Layout', 'Metadata', 'Settings', 'Version')) + { + if ($entries -notcontains $required) { $issues.Add("is missing the $required part. Save it again from Power BI Desktop.") } + } + if ($issues.Count -gt 0) { return $issues } + + # A PBIX saved without loading data has a tiny data model + $dataModelSize = Get-ArchiveEntrySize $Path 'DataModel' + if ($dataModelSize -lt 1MB) + { + $issues.Add("has a $(Format-Size $dataModelSize) data model, so it was saved without loading demo data. Refresh the report before saving.") + } + + $layout = $null + try { $layout = ConvertFrom-PbixJson (Get-ArchiveEntry $Path 'Report/Layout') } + catch { $issues.Add("has a Report/Layout part that couldn't be read ($($_.Exception.Message)).") } + + if ($layout) + { + # Version placeholders are replaced at build time, so finding one means the PBIX was + # saved from the source project instead of the generated project in release/pbix + $sections = @($layout.sections) + $displayOrder = @($sections | Sort-Object @{ Expression = { if ($null -eq $_.ordinal) { 0 } else { $_.ordinal } } }) + $expected = [array]::FindIndex($displayOrder, [Predicate[object]] { param($section) $section.displayName -eq 'Get started' }) + + $config = $layout.config | ConvertFrom-Json -Depth 100 + if ($expected -ge 0 -and $config.activeSectionIndex -ne $expected) + { + $active = if ($config.activeSectionIndex -lt $displayOrder.Count) { $displayOrder[$config.activeSectionIndex].displayName } else { "page $($config.activeSectionIndex)" } + $issues.Add("opens on '$active'. Switch to the Get started page and save again.") + } + + $layoutText = ($layout | ConvertTo-Json -Depth 100 -Compress) + if ($layoutText -match '\$\$ftkver\$\$|\$\$build-date\$\$') + { + $issues.Add("still has version placeholders, so it was saved from src/power-bi instead of release/pbix. Open the project from release/pbix and save again.") + } + elseif ($layoutText -notmatch [regex]::Escape($version)) + { + $issues.Add("doesn't mention version $version, so it was saved from an older build. Rebuild and save again.") + } + } + + # A report saved from the shared project carries every table, not just the ones it needs + if ($entries -contains 'DiagramLayout') + { + try + { + $diagram = ConvertFrom-PbixJson (Get-ArchiveEntry $Path 'DiagramLayout') + $tables = @($diagram.diagrams.nodes | ForEach-Object { $_.nodeIndex } | Where-Object { $_ }) + $extra = @($tables | Where-Object { $Report.tables -notcontains $_ } | Sort-Object -Unique) + if ($extra.Count -gt 0) + { + $issues.Add("includes $($extra.Count) table$(if ($extra.Count -ne 1) { 's' }) the report doesn't use ($($extra -join ', ')). Open the project from release/pbix and save again.") + } + } + catch { Write-Verbose "Could not read DiagramLayout from $Path" } + } + + return $issues +} + +#endregion Helpers + +#region State + +if ($Build -and -not $Status) +{ + Write-Host 'Building Power BI templates and projects...' + & "$PSScriptRoot/Build-PowerBI.ps1" + Write-Host '' +} + +$hasManifest = Test-Path $manifestPath +$manifest = if ($hasManifest) { Get-Content $manifestPath -Raw | ConvertFrom-Json } else { $null } +$isStale = $hasManifest -and $manifest.version -ne $version + +if ((-not $hasManifest -or $isStale) -and -not $Status -and -not $Build) +{ + if ($isStale) { Write-Host "Templates were built for $($manifest.version) but the current version is $version. Rebuilding..." } + else { Write-Host 'No Power BI build found. Building...' } + & "$PSScriptRoot/Build-PowerBI.ps1" + Write-Host '' + $hasManifest = Test-Path $manifestPath + $manifest = if ($hasManifest) { Get-Content $manifestPath -Raw | ConvertFrom-Json } else { $null } + $isStale = $false +} + +if (-not $hasManifest) +{ + Write-Host '⏳ Power BI templates have not been built.' + Write-Host ' Run: ' -NoNewline + Write-Host './Package-PowerBI -Build' -ForegroundColor Cyan + return +} + +# Only storage reports ship as demo PBIX files +$demoReports = @($manifest.reports | Where-Object { $_.type -eq 'storage' }) +$templateZips = @("$relDir/PowerBI-kql.zip", "$relDir/PowerBI-storage.zip") +$demoZip = "$relDir/PowerBI-demo.zip" + +$saved = @() +$missing = @() +foreach ($report in $demoReports) +{ + if (Test-Path "$pbixDir/$($report.pbix)") { $saved += $report } else { $missing += $report } +} + +#endregion State + +#region Report status + +Write-Host "Power BI release files for $version" -ForegroundColor White +Write-Host '' + +$builtTemplates = @(Get-ChildItem "$pbitDir/*.pbit" -ErrorAction SilentlyContinue) +$zipsBuilt = @($templateZips | Where-Object { Test-Path $_ }) +Write-Host " $(if ($builtTemplates.Count -eq $manifest.reports.Count -and $zipsBuilt.Count -eq 2) { '✅' } else { '⏳' }) Templates $($builtTemplates.Count)/$($manifest.reports.Count) PBIT, $($zipsBuilt.Count)/2 ZIP" +Write-Host " $(if ($missing.Count -eq 0) { '✅' } else { '⏳' }) Demo reports $($saved.Count)/$($demoReports.Count) PBIX saved" +Write-Host " $(if (Test-Path $demoZip) { '✅' } else { '⏳' }) Demo package $(if (Test-Path $demoZip) { Format-Size (Get-Item $demoZip).Length } else { 'PowerBI-demo.zip not created' })" +Write-Host '' + +if ($Status) { return } + +#endregion Report status + +#region Save PBIX files + +if ($missing.Count -gt 0) +{ + Write-Host "⏳ $($missing.Count) Power BI project$(if ($missing.Count -ne 1) { 's' }) still $(if ($missing.Count -eq 1) { 'needs' } else { 'need' }) to be saved as PBIX:" + $missing | ForEach-Object { Write-Host " $($_.pbip) → $($_.pbix)" } + Write-Host '' + + if (-not $Open) + { + Write-Host ' To open them, run: ' -NoNewline + Write-Host './Package-PowerBI -Open' -ForegroundColor Cyan + return + } + + Write-Host 'For each project that opens:' + Write-Host ' 1. Refresh the report so demo data is loaded.' + Write-Host ' 2. Select File > Save as, keep the release/pbix folder, and change the file type to PBIX.' + Write-Host ' 3. Set the sensitivity to "Public" when prompted.' + Write-Host ' 4. Switch to the Get started page and save again.' + Write-Host '' + Write-Host 'Queries are already trimmed to what each report needs, so there is nothing to remove by hand.' + Write-Host '' + + if ($IsWindows -or $null -eq $IsWindows) + { + $missing | ForEach-Object { Invoke-Item "$pbixDir/$($_.pbip)" } + Write-Host "Opened $($missing.Count) project$(if ($missing.Count -ne 1) { 's' }) in Power BI Desktop." + } + else + { + Write-Warning 'Power BI Desktop only runs on Windows. Open these projects there:' + $missing | ForEach-Object { Write-Host " $((Resolve-Path "$pbixDir/$($_.pbip)").Path)" } + } + + Write-Host '' + Write-Host ' When they are all saved, run: ' -NoNewline + Write-Host './Package-PowerBI' -ForegroundColor Cyan + return +} + +#endregion Save PBIX files + +#region Validate and package + +Write-Host "Checking $($saved.Count) demo report$(if ($saved.Count -ne 1) { 's' })..." +$failed = 0 +foreach ($report in $saved) +{ + $path = "$pbixDir/$($report.pbix)" + $issues = Test-DemoPbix $path $report + if ($issues.Count -eq 0) + { + Write-Host " ✅ $($report.pbix) $(Format-Size (Get-Item $path).Length)" + } + else + { + $failed++ + Write-Host " ❌ $($report.pbix)" -ForegroundColor Red + $issues | ForEach-Object { Write-Host " $_" -ForegroundColor Red } + } +} + +if ($failed -gt 0) +{ + Write-Host '' + throw "$failed demo report$(if ($failed -ne 1) { 's' }) failed validation. Fix the issues above, save again, and rerun this command." +} + +Write-Host '' +Write-Host 'Packaging PowerBI-demo.zip...' + +# PBIX files are already compressed, so the fastest level takes far less time for the same size +Remove-Item $demoZip -Force -ErrorAction SilentlyContinue +Compress-Archive -Path "$pbixDir/*.storage.pbix" -DestinationPath $demoZip -CompressionLevel Fastest + +Write-Host '' +Write-Host "✅ Power BI release files for $version" -ForegroundColor Green +@($templateZips + $demoZip) ` +| Where-Object { Test-Path $_ } ` +| ForEach-Object { Write-Host " $(Split-Path $_ -Leaf) $(Format-Size (Get-Item $_).Length)" } + +#endregion Validate and package diff --git a/src/scripts/Package-Toolkit.ps1 b/src/scripts/Package-Toolkit.ps1 index 80178a440..d1cdd93e4 100644 --- a/src/scripts/Package-Toolkit.ps1 +++ b/src/scripts/Package-Toolkit.ps1 @@ -18,10 +18,10 @@ Optional. Indicates whether to copy templates and open data files. Default = false. .PARAMETER OpenPBI - Optional. Indicates that Power BI storage projects should be opened so they can be manually saved as *.storage.pbix files. Default = false. + Optional. Opens the generated Power BI projects so they can be saved as *.storage.pbix files. Equivalent to Package-PowerBI -Open. Default = false. .PARAMETER ZipPBI - Optional. Indicates that demo PBIX files should be packaged into PowerBI-demo.zip. Default = false. + Optional. Validates the saved PBIX files and packages them into PowerBI-demo.zip. Equivalent to Package-PowerBI. Default = false. .PARAMETER Preview Optional. Indicates that the template(s) should be saved as a preview only. Does not package other files. Default = false. @@ -44,7 +44,7 @@ .EXAMPLE ./Package-Toolkit -ZipPBI - Generates the PowerBI-demo.zip file. Must be run after -OpenPBI. + Validates the saved PBIX files and generates the PowerBI-demo.zip file. Must be run after -OpenPBI. #> param( [Parameter(Position = 0)][string]$Template = "*", @@ -88,7 +88,9 @@ function Copy-TemplateFiles() if ($Template -eq "*") { Write-Verbose "Removing existing ZIP files..." - Remove-Item "$relDir/*.zip" -Force + # Power BI ZIP files are managed by Build-PowerBI and Package-PowerBI. Deleting them here + # would throw away the templates that -Build just generated. + Remove-Item "$relDir/*.zip" -Force -Exclude 'PowerBI-*.zip' } return Get-ChildItem "$relDir/$Template*" -Directory ` @@ -300,74 +302,20 @@ if ($CopyFiles -or $Build -or $Preview -or -not ($OpenPBI -or $ZipPBI)) } } -# Open or zip Power BI storage reports to generate PowerBI-demo.zip -$pbip = Get-ChildItem -Path "$PSScriptRoot/../power-bi/storage" -Include "*.pbip" -Recurse -if (-not ($OpenPBI -or $ZipPBI)) +# Power BI files are packaged by Package-PowerBI, which reports what's left to do +if ($OpenPBI) { - Write-Host "⚠️ $($pbip.Count) Power BI projects must be manually saved as *.storage.pbix!" - Write-Host ' To open them, run: ' -NoNewline - Write-Host './Package-Toolkit -OpenPBI' -ForegroundColor Cyan + & "$PSScriptRoot/Package-PowerBI.ps1" -Open } -elseif ($OpenPBI) +elseif ($ZipPBI) { - # Create temp pbix folder and remove existing storage/demo reports - & "$PSScriptRoot/New-Directory" "$relDir/pbix" - Remove-Item -Path "$relDir/pbix/*.storage.pbix" -Recurse -Force -ErrorAction SilentlyContinue - Remove-Item -Path "$relDir/pbix/*.demo.pbix" -Recurse -Force -ErrorAction SilentlyContinue - - # Open Power BI projects - Write-Host "ℹ️ $($pbip.Count) Power BI projects must be saved as PBIX first... Opening..." - $pbip | Invoke-Item - Write-Host ' Save as *.storage.pbix then run: ' -NoNewline - Write-Host './Package-Toolkit -ZipPBI' -ForegroundColor Cyan + & "$PSScriptRoot/Package-PowerBI.ps1" } -elseif ($ZipPBI) +elseif (-not $Preview) { - $pbixFiles = Get-ChildItem -Path "$relDir/pbix/*.storage.pbix" - - # TODO: Confirm all files are ready - if ($pbip.Count -ne $pbixFiles.Count) - { - Write-Host "⚠️ Found $($pbip.Count) Power BI projects but $($pbixFiles.Count) *.storage.pbix files. Please confirm all projects are saved as PBIX files first." - return - } - - # Clean PBIX files - Write-Verbose "Processing $($pbixFiles.Count) *.storage.pbix files..." - $pbixFiles | ForEach-Object { - # Expand PBIX files for cleanup - $pbix = $_ - $pbixDir = $pbix.FullName.Replace('.storage.pbix', '.demo') - Write-Verbose "Expanding $pbixDir..." - Remove-Item -Path $pbixDir -Recurse -Force -ErrorAction SilentlyContinue - Expand-Archive -Path $pbix -DestinationPath $pbixDir - - # Remove _rels, docProps - Remove-Item -Path "$pbixDir/_rels" -Recurse -Force -ErrorAction SilentlyContinue - Remove-Item -Path "$pbixDir/docProps" -Recurse -Force -ErrorAction SilentlyContinue - - # Remove docProps from Content_Types - $contentTypes = [xml](Get-Content "$pbixDir/?Content_Types?.xml" -Raw) - $contentTypes.Types.Override | Where-Object { - $_.PartName -eq '/docProps/custom.xml' } | ForEach-Object { $contentTypes.Types.RemoveChild($_) | Out-Null - } - $contentTypes.Save("$pbixDir/[Content_Types].xml") | Out-Null - - # Remove unneeded tables - # $diagramLayout = Get-Content "$pbixDir/DiagramLayout" -Raw | ConvertFrom-Json -Depth 100 - # $diagramLayout.diagrams.nodes | Where-Object { $metadata.Tables -contains $_.nodeIndex } - # TODO: Save as UTF-16LE -- $diagramLayout | ConvertTo-Json -Depth 100 - - # TODO: Remove tables from data model - # pbi-tools extract $pbix - # TODO: Remove tables - # pbi-tools compile $pbixDir $pbix - - # Zip as PBIX for demo - Write-Verbose "Saving demo $pbixDir.pbix..." - Remove-Item -Path "$pbixDir.pbix" -Recurse -Force -ErrorAction SilentlyContinue - Compress-Archive -Path $pbixDir -DestinationPath "$pbixDir.pbix" - } + & "$PSScriptRoot/Package-PowerBI.ps1" -Status + Write-Host ' To continue, run: ' -NoNewline + Write-Host './Package-PowerBI' -ForegroundColor Cyan } Write-Host '...done!' diff --git a/src/scripts/README.md b/src/scripts/README.md index 6aeb2c5ad..52742163c 100644 --- a/src/scripts/README.md +++ b/src/scripts/README.md @@ -14,6 +14,8 @@ On this page: - [🏷️ Get-Version](#️-get-version) - [🏷️ Update-Version](#️-update-version) - [🚚 Publish-Toolkit](#-publish-toolkit) +- [📊 Build-PowerBI](#-build-powerbi) +- [📊 Package-PowerBI](#-package-powerbi) - [📦 Package-Toolkit](#-package-toolkit) - [©️ Add-CopyrightHeader](#️-add-copyrightheader) - [📁 New-Directory](#-new-directory) @@ -454,6 +456,70 @@ Examples:
+## 📊 Build-PowerBI + +[Build-PowerBI.ps1](./Build-PowerBI.ps1) generates the Power BI release artifacts: + +- One PBIT template per report in `release/pbit`, zipped into `PowerBI-kql.zip` and `PowerBI-storage.zip`. +- One PBIP project per report in `release/pbix`, containing only the tables, relationships, and queries that report needs. + +Both come from a single prune, so the template and the demo report always match. Open the generated project from `release/pbix` to save a demo PBIX — there are no queries to remove by hand. + +Templates ship with the data source parameters set to null. The generated projects keep the demo values so demo reports can still refresh. + +| Parameter | Description | +| ----------- | ---------------------------------------------------------------------------------------------------------------------- | +| `‑Name` | Optional. Name of the report to build. Wildcards supported. Default = \* (all). | +| `‑KQL` | Optional. Builds the KQL reports. Default = false (builds all if no types are selected). | +| `‑Storage` | Optional. Builds the storage reports. Default = false (builds all if no types are selected). | +| `‑NoPbip` | Optional. Skips generating PBIP projects and only builds PBIT templates. Default = false. | + +Examples: + +- Generate all templates and projects. + + ```powershell + ./Build-PowerBI + ``` + +- Generate the Cost summary storage template and project. + + ```powershell + ./Build-PowerBI CostSummary -Storage + ``` + +
+ +## 📊 Package-PowerBI + +[Package-PowerBI.ps1](./Package-PowerBI.ps1) packages the three Power BI release files and reports what's left to do. + +Power BI Desktop has to load and save the demo PBIX files, so this command is resumable: run it, save the projects it opens, then run it again. It works out which steps are already done, does the next one, and validates the result. + +Saved PBIX files are checked for the mistakes that are easy to make by hand — saved without data, saved from the unpruned source project, saved on the wrong page, or saved from a stale build — so a missed step fails here instead of shipping. + +| Parameter | Description | +| ---------- | ---------------------------------------------------------------------------------------------- | +| `‑Open` | Optional. Opens the projects that still need to be saved as PBIX files. Default = false. | +| `‑Build` | Optional. Rebuilds the templates and projects even if they already exist. Default = false. | +| `‑Status` | Optional. Reports what's done and what's left without changing anything. Default = false. | + +Examples: + +- Build whatever is missing and report the next step. + + ```powershell + ./Package-PowerBI + ``` + +- Open the projects that still need to be saved as PBIX files. + + ```powershell + ./Package-PowerBI -Open + ``` + +
+ ## 📦 Package-Toolkit [Package-Toolkit.ps1](./Package-Toolkit.ps1) packages all toolkit templates as ZIP files for release. @@ -462,7 +528,9 @@ Examples: | ----------- | -------------------------------------------------------------------------------------------------------------------------- | | `‑Template` | Optional. Name of the template or module to package. Default = \* (all). | | `‑Build` | Optional. Indicates whether the Build-Toolkit command should be executed first. Default = false. | -| `‑PowerBI` | Optional. Indicates whether to open Power BI files as part of the packaging process. Default = false. | +| `‑CopyFiles` | Optional. Indicates whether to copy templates and open data files. Default = false. | +| `‑OpenPBI` | Optional. Opens the generated Power BI projects to be saved as PBIX files. Same as `Package-PowerBI -Open`. Default = false. | +| `‑ZipPBI` | Optional. Validates the saved PBIX files and packages PowerBI-demo.zip. Same as `Package-PowerBI`. Default = false. | | `‑Preview` | Optional. Indicates that the template(s) should be saved as a preview only. Does not package other files. Default = false. | Examples: From 0602bf0c068ac9c7479574d1961b4f47bd0cd957 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Sat, 12 Sep 2026 05:08:09 +0000 Subject: [PATCH 02/50] chore: Update ms.date in docs-mslearn files --- docs-mslearn/toolkit/changelog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 6cb44efdf..310562a2b 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 09/09/2026 +ms.date: 09/12/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit From de45d64a143d6ada7ebc7080469468a2de563929 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sun, 20 Sep 2026 16:54:57 -0700 Subject: [PATCH 03/50] feat(power-bi): automate demo PBIX saves and validate release artifacts Adds Save-PowerBIProject, which drives Power BI Desktop through UI Automation to open a generated project, refresh it through Desktop's local Analysis Services engine, apply the sensitivity label, save it as PBIX, and close. Package-PowerBI -Unattended runs it for every demo report, then validates and packages, so the whole Power BI release is one command on Windows. Failures name the step and save a screenshot, and -Open is still there as the manual fallback. Moves the per-report table and query allowlist to src/power-bi/reports.json so the build and the tests read the same list, and adds checks that fail with the report, page, visual, and what to add: - A visual on a visible page reads from a table the report doesn't keep. Hidden pages warn instead. - A query, table, or measure the report keeps uses a table or query the report doesn't keep. A new PowerBIReports lint test runs these checks in pull request CI, so a mismatch fails the PR that causes it instead of surfacing at release time. Build-PowerBI.Tests.ps1 covers pruning, TMDL parsing, name conflicts, dependency and reference detection, and every PBIX validation rule without needing the Analysis Services library or Power BI Desktop. Fixes found by the new checks, five of which shipped in v14: - The Governance report's Virtual machines page read from VirtualMachinesDisks, which the allowlist removed, so the disks table has been broken since #1647. - AdvisorReservationRecommendations declared Region and region, which a tabular model can't hold, so Build-PowerBI failed on dev. Both models now have one region column that coalesces either casing from Azure Resource Graph. - A storage relationship referenced AdvisorReservationRecommendations.subscriptionId, a column that never existed, so the full storage model couldn't load. - The data source nulling regex expected an escaped quote that was never there, so every shipped template had the demo storage account and cluster prefilled. - PBI_QueryOrder was double JSON encoded and leaked literal [storage] and [kql] prefixes. - DiagramLayout was never pruned, so templates listed tables the model no longer had. - Governance listed PolicyDefinitions under tables, but it's a query, so it was dropped from the template. - Package-Toolkit -Build -CopyFiles deleted the Power BI ZIP files that -Build had just generated. - -ZipPBI never created PowerBI-demo.zip, and its cleanup nested every PBIX part one folder deep. Confirmed unused (v14 ships the raw PBIX files) and removed. Build-PowerBI also reported errors but exited 0 with truncated PBIT files. It now fails loudly, and drops tables no report ships before loading the model so an unused broken table can't block a release. Co-Authored-By: Claude Opus 5 --- .claude/commands/release.md | 14 +- .../-internal-only--release-checklist.md | 16 +- .github/workflows/dev.yml | 6 + docs-mslearn/toolkit/changelog.md | 3 +- docs-wiki/Build-and-test.md | 6 +- docs-wiki/Release-process.md | 25 +- src/power-bi/README.md | 4 + .../AdvisorReservationRecommendations.tmdl | 13 +- src/power-bi/reports.json | 172 +++++++ .../definition/relationships.tmdl | 4 - .../AdvisorReservationRecommendations.tmdl | 13 +- .../Tests/Lint/PowerBIReports.Tests.ps1 | 83 +++ .../Tests/Unit/Build-PowerBI.Tests.ps1 | 371 ++++++++++++++ src/scripts/Build-PowerBI.ps1 | 466 ++++++++++++----- src/scripts/Package-PowerBI.ps1 | 198 ++++++-- src/scripts/Package-Toolkit.ps1 | 4 +- src/scripts/README.md | 77 ++- src/scripts/Save-PowerBIProject.ps1 | 472 ++++++++++++++++++ 18 files changed, 1696 insertions(+), 251 deletions(-) create mode 100644 src/power-bi/reports.json create mode 100644 src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 create mode 100644 src/powershell/Tests/Unit/Build-PowerBI.Tests.ps1 create mode 100644 src/scripts/Save-PowerBIProject.ps1 diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 437de3337..c3b1118cd 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -121,7 +121,7 @@ Present as a prioritized list — no AUQ needed, just a summary the user can act Update the release tracking issue checkboxes: 1. Run `gh issue view {number} --json body` and capture the output. Parse the JSON to extract the `body` field, then write it to `/tmp/release-issue-body.md` using the Write tool (not shell redirection). -2. Read the file with the Read tool. Each `/release`-managed checkbox in the template has an HTML comment label like ``, ``, or ``. To mark a step complete, find the line containing the matching label and change its `- [ ]` to `- [x]`. Use the Edit tool to write the updated body back to the file. In Phase 2 Release readiness, mark `release:core` complete. +2. Read the file with the Read tool. Each `/release`-managed checkbox in the template has an HTML comment label like ``, ``, ``, or ``. To mark a step complete, find the line containing the matching label and change its `- [ ]` to `- [x]`. Use the Edit tool to write the updated body back to the file. In Phase 2 Release readiness, mark `release:core` complete. 3. Run `gh issue edit {number} --body-file /tmp/release-issue-body.md` to push the updates. Then present a summary and next action via AskUserQuestion: @@ -150,13 +150,17 @@ If it fails, show the error and ask whether to investigate or skip. ### Package Power BI -Power BI packaging is resumable and reports its own state. Run: +Power BI packaging reports its own state and saves demo reports with Power BI Desktop, which only runs on Windows. + +On Windows, tell the user not to use the mouse or keyboard until it finishes, then run: ```bash -pwsh -Command "./src/scripts/Package-PowerBI.ps1" +pwsh -Command "./src/scripts/Package-PowerBI.ps1 -Unattended" ``` -Relay its status output verbatim — it says exactly which step is next. When it reports projects that still need to be saved, tell the user to run `Package-PowerBI.ps1 -Open`, save each project as PBIX (refresh, Save as PBIX in `release/pbix`, sensitivity "Public", end on the Get started page), then say "done" so you can rerun the command to validate and package. +On other platforms, run `pwsh -Command "./src/scripts/Package-PowerBI.ps1 -Status"` and tell the user to run `Package-PowerBI.ps1 -Unattended` on a Windows machine with Power BI Desktop, then say "done". + +Relay its status output verbatim — it says exactly which step is next. If a report can't be saved automatically, relay the failed step and screenshot path, tell the user to run `Package-PowerBI.ps1 -Open` and save that project as PBIX (refresh, Save as PBIX in `release/pbix`, sensitivity "Public"), then say "done" so you can rerun the command to validate and package. If validation fails, show the reported issues verbatim. Each one names the file and the fix; don't guess at causes. @@ -171,4 +175,4 @@ After packaging succeeds, inform the user of remaining manual steps documented i ### Final issue update -Update the release tracking issue checkboxes for finalize-phase items using the same label-based process as Release readiness. Mark `release:finalize` and `release:package` complete. Then report completion and wish the user well. +Update the release tracking issue checkboxes for finalize-phase items using the same label-based process as Release readiness. Mark `release:finalize` and `release:package` complete. Mark `release:powerbi` complete only when all three Power BI files exist. Then report completion and wish the user well. diff --git a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md index 7ad3d862c..7014d4a42 100644 --- a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md +++ b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md @@ -110,16 +110,12 @@ Status icons: > _This step is optional, but can catch issues earlier. You can also add the `-Build` parameter to the publish command in the next step._ - Ensure all tests pass: `/src/scripts/Test-PowerShell -Unit -Integration` - [ ] Package all release files (except Power BI): `/src/scripts/Package-Toolkit.ps1 -Build -CopyFiles` script -- [ ] Package Power BI files: `/src/scripts/Package-PowerBI.ps1` - > _This command is resumable. Run it, do what it asks, then run it again. It reports what's done, what's left, and checks the saved PBIX files for missed steps._ - - [ ] Run `Package-PowerBI.ps1 -Open` to open the projects that need to be saved. - - [ ] Save and close each Power BI project: - - Refresh the report so demo data is loaded. - - Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. - - When prompted, set the sensitivity to "Public". - - Verify all pages, switch to the Get started page, and save again. - > _Queries are already trimmed to what each report needs. There is nothing to remove by hand._ - - [ ] Run `Package-PowerBI.ps1` to validate the saved files and create PowerBI-demo.zip. +- [ ] Package Power BI files on Windows: `/src/scripts/Package-PowerBI.ps1 -Unattended` + > _Builds, saves demo PBIX files with Power BI Desktop, validates, and packages. Don't use the mouse or keyboard while it runs. Sign in to Power BI Desktop first so the "Public" sensitivity label can be applied._ + - If a report can't be saved automatically, the command names the step that failed and saves a screenshot. Save that report by hand, then rerun the command: + - Run `Package-PowerBI.ps1 -Open` to open the projects that need to be saved. + - Refresh the report, select **File** > **Save as**, keep the `/release/pbix` folder, change the file type to PBIX, and set the sensitivity to "Public". + > _Queries are already trimmed and the Get started page is already selected. There is nothing else to change._ - [ ] Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. - [ ] Check the docs for broken links: - Create a personal fork of the main repo. diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 1702ee02d..7fd648f3b 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -11,6 +11,12 @@ on: - 'src/templates/finops-hub/**/timeZones.bicep' - 'docs/deploy/finops-hub-latest.json' - 'docs/deploy/finops-hub-preview.json' + # Power BI report definitions and build scripts are covered by PowerBIReports.Tests.ps1 and Build-PowerBI.Tests.ps1 + - 'src/power-bi/kql/**' + - 'src/power-bi/storage/**' + - 'src/power-bi/reports.json' + - 'src/scripts/Build-PowerBI.ps1' + - 'src/scripts/Package-PowerBI.ps1' jobs: run_pester_tests: name: Pester diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 6cb44efdf..d48bef7ba 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 09/09/2026 +ms.date: 09/20/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -130,6 +130,7 @@ _Released June 2026_ - Updated report labels and documentation to use current FinOps Framework capability names, including Usage optimization and Governance, policy, and risk ([#2170](https://github.com/microsoft/finops-toolkit/pull/2170)). - **Fixed** - Fixed report templates opening with a preconfigured demo data source. Storage and cluster settings are now empty when you open a template. + - Fixed the managed disks table on the **Virtual machines** page of the [Governance](power-bi/governance.md) report, which showed an error because the report template didn't include the disks data. - Fixed Power BI storage report refresh errors caused by ISO 8601 duration `x_SkuTerm` values (like `P3Y`) and empty strings in cost exports ([#2174](https://github.com/microsoft/finops-toolkit/issues/2174)). - Paginate Azure Resource Graph queries by subscription to mitigate [payload size limit](help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) errors in the [Governance](power-bi/governance.md) and [Workload optimization](power-bi/workload-optimization.md) reports ([#1768](https://github.com/microsoft/finops-toolkit/issues/1768)). Queries still surface a payload size error (rather than silently returning truncated results) if a single batch of subscriptions exceeds the limit; see [Reduce the batch size](help/errors.md#option-1-reduce-the-batch-size) for the mitigation. As part of this change, the `AdvisorRecommendations` and `AdvisorReservationRecommendations` tables now return complete (non-truncated) results per batch, matching every other batched table; previously these two tables silently dropped rows beyond the payload limit instead of erroring. - Fixed the [Governance](power-bi/governance.md) report template missing the policy definitions query. diff --git a/docs-wiki/Build-and-test.md b/docs-wiki/Build-and-test.md index 9154d1802..d220370d3 100644 --- a/docs-wiki/Build-and-test.md +++ b/docs-wiki/Build-and-test.md @@ -142,9 +142,11 @@ cd /src/scripts This generates two things from a single prune, so the template and the demo report can't drift: - PBIT templates in `release/pbit`, zipped into `PowerBI-kql.zip` and `PowerBI-storage.zip`. -- PBIP projects in `release/pbix`, each containing only the tables, relationships, and queries that report needs. +- PBIP projects for demo reports in `release/pbix`, each containing only the tables, relationships, and queries that report needs. -Power BI Desktop has to load and save the demo PBIX files, so that step is still manual. Use [Package-PowerBI](../src/scripts/README.md#-package-powerbi), which reports what's left to do and validates the saved files. +The tables and queries each report keeps are listed in [src/power-bi/reports.json](../src/power-bi/reports.json). When a report starts using a new table or query, add it there. The build and the `PowerBIReports` lint test fail when a visual or query uses something the report doesn't keep. + +Demo PBIX files need Power BI Desktop to load and save them. On Windows, run [Package-PowerBI](../src/scripts/README.md#-package-powerbi) `-Unattended` to save, validate, and package them automatically. ### Building documentation diff --git a/docs-wiki/Release-process.md b/docs-wiki/Release-process.md index 404c88f61..ce9ee1f4d 100644 --- a/docs-wiki/Release-process.md +++ b/docs-wiki/Release-process.md @@ -121,25 +121,26 @@ Once the above requirements have been met, the feature branch can be merged into > _The documentation site may take 5 minutes to update after the merge is committed. If not updated, look at [GitHub actions](../actions/workflows/pages/pages-build-deployment) to see if there are any failures._ - 6. Package the Power BI files: + 6. Package the Power BI files on Windows with Power BI Desktop installed: ```powershell cd /src/scripts - ./Package-PowerBI + ./Package-PowerBI -Unattended ``` - > _This command is resumable. Run it, do what it asks, then run it again. It reports what's done, what's left, and checks the saved PBIX files for missed steps._ + > _Builds the templates, saves each demo report as a PBIX file with Power BI Desktop, validates the files, and packages them. Don't use the mouse or keyboard while it runs. Sign in to Power BI Desktop first so the **Public** sensitivity label can be applied._ + > + > ⚠️ _If the sensitivity label isn't **Public**, the report will not open for anyone outside of Microsoft. Validation fails when a demo report has any other label._ + + If a report can't be saved automatically, the command names the step that failed and saves a screenshot next to the PBIX file. Save that report by hand, then rerun the command: 1. Run `./Package-PowerBI -Open` to open the projects that still need to be saved. - 2. For each Power BI project that opens: - 1. Refresh the report so demo data is loaded. - 2. Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. - 3. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠️ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - 4. Verify all pages, switch to the **Get started** page, and save again. - > _Queries are already trimmed to what each report needs. There is nothing to remove by hand, and the version is already stamped._ - 3. Run `./Package-PowerBI` again to validate the saved files and create PowerBI-demo.zip. - 4. Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. + 2. Refresh the report so demo data is loaded. + 3. Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. + 4. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. + > _Queries are already trimmed and the **Get started** page is already selected. There is nothing else to change._ + + Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. 7. Tag and publish a [new release](../releases/new): 1. Create a tag on publish using the "vX.X" format. 2. Set the **Target** to `main`. diff --git a/src/power-bi/README.md b/src/power-bi/README.md index 9cbc3ca4c..7df8f9fe5 100644 --- a/src/power-bi/README.md +++ b/src/power-bi/README.md @@ -65,6 +65,10 @@ Power BI reports in the FinOps toolkit must adhere to the following guidelines: 2. Queries should all be a the root **FinOps toolkit** folder. 1. All queries should use the same settings. Do not duplicate properties that can be shared. 7. All text (including report, page, and column names) should follow the [content guidelines](../../docs-wiki/Coding-guidelines.md#-content-strings-and-microcopy). +8. Every table and query a report uses must be listed for that report in [reports.json](./reports.json). + 1. Reports share one semantic model per data source. The release build removes everything a report doesn't list, so an unlisted table shows up as a broken visual or refresh error. + 2. Prefix a name with `[kql]` or `[storage]` when only one report type uses it. + 3. The `PowerBIReports` lint test fails when a visual or query uses something the report doesn't list. > [!NOTE] > Remember: These are general rules of thumb for consistency. They are not set in stone. Please propose updates by submitting a PR, starting a discussion, or creating an issue. In general, use your best judgement to make the report as useful as possible. We just want everything to feel consistent and familiar. diff --git a/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl b/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl index 2dab7934a..5de53c619 100644 --- a/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl +++ b/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl @@ -380,14 +380,6 @@ table AdvisorReservationRecommendations annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.Region' - dataType: string - lineageTag: 68593266-9950-4caf-9522-ccfcbaa5d5e3 - summarizeBy: none - sourceColumn: properties.extendedProperties.Region - - annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.commitment' dataType: string lineageTag: e0b87d0c-8c42-4354-b85a-5fc93d4af06a @@ -469,11 +461,10 @@ table AdvisorReservationRecommendations ['properties.extendedProperties.DatabaseSize'] = properties.extendedProperties.DatabaseSize, ['properties.extendedProperties.DatabaseName'] = properties.extendedProperties.DatabaseName, ['properties.extendedProperties.ServerName'] = properties.extendedProperties.ServerName, - ['properties.extendedProperties.Region'] = properties.extendedProperties.Region, ['properties.extendedProperties.displaySKU'] = properties.extendedProperties.displaySKU, ['properties.extendedProperties.displayQty'] = toint(properties.extendedProperties.displayQty), ['properties.extendedProperties.location'] = properties.extendedProperties.location, - ['properties.extendedProperties.region'] = properties.extendedProperties.region, + ['properties.extendedProperties.region'] = coalesce(tostring(properties.extendedProperties.region), tostring(properties.extendedProperties.Region)), ['properties.extendedProperties.vmSize'] = properties.extendedProperties.vmSize, ['properties.extendedProperties.subId'] = properties.extendedProperties.subId, ['properties.extendedProperties.scope'] = properties.extendedProperties.scope, @@ -496,7 +487,7 @@ table AdvisorReservationRecommendations NullHandling = if Source = null then #table( - { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, + { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, {} ) else Source, diff --git a/src/power-bi/reports.json b/src/power-bi/reports.json new file mode 100644 index 000000000..de294f101 --- /dev/null +++ b/src/power-bi/reports.json @@ -0,0 +1,172 @@ +{ + "$comment": [ + "Build settings for each Power BI report, used by src/scripts/Build-PowerBI.ps1.", + "description: Template description shown when the PBIT file is opened.", + "tables: Tables to keep. Everything else is removed from the template and demo project.", + "queries: Parameters, functions, and queries that do not load a table (Power Query expressions) to keep.", + "Prefix a name with [kql] or [storage] to keep it for that report type only." + ], + "reports": { + "CostSummary": { + "description": "The Cost summary report provides several summaries of your effective (amortized) and billed costs based on the FinOps Open Cost and Usage Specification (FOCUS). Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit. Effective costs will not match your invoice.", + "tables": ["Costs", "Prices", "PricingUnits"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "DataIngestion": { + "description": "The Data ingestion report provides details about the data you've ingested into your FinOps hub storage account.", + "tables": ["Costs", "HubScopes", "HubSettings", "Prices", "PricingUnits", "StorageData", "StorageErrors"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "Governance": { + "description": "The Governance, policy, and risk report summarizes your Microsoft Cloud governance posture. It offers the standard metrics aligned with the Cloud Adoption Framework to facilitate identifying issues, applying recommendations, and resolving compliance gaps.", + "tables": [ + "AdvisorRecommendations", + "Compliance calculation", + "Costs", + "Disks", + "ManagementGroups", + "NetworkInterfaces", + "NetworkSecurityGroups", + "PolicyAssignments", + "PolicyStates", + "Prices", + "PricingUnits", + "PublicIPAddresses", + "Regions", + "Resources", + "ResourceTypes", + "SqlDatabases", + "Subscriptions", + "VirtualMachines", + "VirtualMachinesDisks" + ], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "PolicyDefinitions", + "Remove Duplicate Resource IDs", + "Deprecated: Perform Extra Query Optimizations", + "ftk_ARGBatchSize", + "ftk_DemoFilter", + "ftk_QueryARG", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "Invoicing": { + "description": "The Invoicing and chargeback report provides several summaries of your billed cost to facilitate invoice reconciliation for Microsoft Customer Agreement (MCA) and Enterprise Agreement (EA) accounts or to perform chargeback using effective (amortized) costs.", + "tables": ["Costs", "Prices", "PricingUnits"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "RateOptimization": { + "description": "The Rate optimization report provides insights into any rate optimization opportunities, like reservations, savings plans, and Azure Hybrid Benefit. This report uses effective cost, which amortizes and breaks reservation and savings plan purchases down and allocates costs out to the resources that received the benefit. Effective cost will not match your invoice.", + "tables": ["Costs", "InstanceSizeFlexibility", "Prices", "PricingUnits", "ReservationRecommendations"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "WorkloadOptimization": { + "description": "The Usage optimization report provides insights into resource utilization and efficiency opportunities based on historical usage patterns. Use this report to determine if resources can be scaled down or even shut down during off-peak hours to minimize wasteful usage and spending. Also consider cheaper alternatives when available and ensure all workloads have some direct or indirect link to business value to avoid unnecessary usage and costs that don't contribute to the mission.", + "tables": ["AdvisorRecommendations", "Costs", "Disks", "Prices", "PricingUnits", "Resources", "Subscriptions"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Remove Duplicate Resource IDs", + "Deprecated: Perform Extra Query Optimizations", + "ftk_ARGBatchSize", + "ftk_DemoFilter", + "ftk_QueryARG", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + } + } +} diff --git a/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl b/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl index 047210e6a..0d85eb1e3 100644 --- a/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl +++ b/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl @@ -64,10 +64,6 @@ relationship 2ad9b8bc-59fb-e94c-a36f-1fa66aaa02a8 fromColumn: AdvisorRecommendations.'properties.resourceMetadata.resourceId' toColumn: Resources.id -relationship AutoDetected_d442ee36-a039-47be-83cf-fd058a0973d7 - fromColumn: AdvisorReservationRecommendations.subscriptionId - toColumn: Subscriptions.subscriptionId - relationship AutoDetected_19cbab3d-1150-45a4-9258-ca0b47ba687d isActive: false fromColumn: AdvisorRecommendations.subscriptionId diff --git a/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl b/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl index 2dab7934a..5de53c619 100644 --- a/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl +++ b/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl @@ -380,14 +380,6 @@ table AdvisorReservationRecommendations annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.Region' - dataType: string - lineageTag: 68593266-9950-4caf-9522-ccfcbaa5d5e3 - summarizeBy: none - sourceColumn: properties.extendedProperties.Region - - annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.commitment' dataType: string lineageTag: e0b87d0c-8c42-4354-b85a-5fc93d4af06a @@ -469,11 +461,10 @@ table AdvisorReservationRecommendations ['properties.extendedProperties.DatabaseSize'] = properties.extendedProperties.DatabaseSize, ['properties.extendedProperties.DatabaseName'] = properties.extendedProperties.DatabaseName, ['properties.extendedProperties.ServerName'] = properties.extendedProperties.ServerName, - ['properties.extendedProperties.Region'] = properties.extendedProperties.Region, ['properties.extendedProperties.displaySKU'] = properties.extendedProperties.displaySKU, ['properties.extendedProperties.displayQty'] = toint(properties.extendedProperties.displayQty), ['properties.extendedProperties.location'] = properties.extendedProperties.location, - ['properties.extendedProperties.region'] = properties.extendedProperties.region, + ['properties.extendedProperties.region'] = coalesce(tostring(properties.extendedProperties.region), tostring(properties.extendedProperties.Region)), ['properties.extendedProperties.vmSize'] = properties.extendedProperties.vmSize, ['properties.extendedProperties.subId'] = properties.extendedProperties.subId, ['properties.extendedProperties.scope'] = properties.extendedProperties.scope, @@ -496,7 +487,7 @@ table AdvisorReservationRecommendations NullHandling = if Source = null then #table( - { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, + { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, {} ) else Source, diff --git a/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 b/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 new file mode 100644 index 000000000..50330e364 --- /dev/null +++ b/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 @@ -0,0 +1,83 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + Lint rule: Power BI reports and src/power-bi/reports.json must agree. + + Build-PowerBI removes every table and query that reports.json doesn't list for a report. When a + visual reads from a table that isn't listed, the template and demo report ship with a broken + visual (for example, the Governance report's "Virtual machines" page lost its disks table this + way). Build-PowerBI fails on this too, but it only runs at release time. This test catches it + in the pull request that causes it. + + Uses the same helpers as Build-PowerBI and only reads files, so it doesn't need the Analysis + Services library. +#> + +BeforeDiscovery { + $powerBiDir = Join-Path (Get-Item -Path $PSScriptRoot).Parent.Parent.Parent.FullName 'power-bi' + $reportProjects = @(Get-ChildItem $powerBiDir -Recurse -Include '*.kql.pbip', '*.storage.pbip' | ForEach-Object { + @{ Name = $_.Name.Split('.')[0]; Type = $_.Name.Split('.')[1]; Folder = $_.DirectoryName; Label = $_.BaseName } + }) +} + +BeforeAll { + $srcDir = (Get-Item -Path $PSScriptRoot).Parent.Parent.Parent.FullName + $powerBiDir = Join-Path $srcDir 'power-bi' + + $ast = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $srcDir 'scripts/Build-PowerBI.ps1'), [ref]$null, [ref]$null) + $ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] }, $false) ` + | ForEach-Object { . ([scriptblock]::Create($_.Extent.Text)) } + + $config = (Get-Content (Join-Path $powerBiDir 'reports.json') -Raw | ConvertFrom-Json -Depth 10).reports + $modelNames = @{} + foreach ($type in 'kql', 'storage') + { + $modelNames[$type] = ConvertTo-NameSet (Get-TmdlObjectName (Join-Path $powerBiDir "$type/Shared.Dataset/definition")) + } +} + +Describe 'PowerBIReports' { + Context '