diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 4243580cb..601686e64 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -121,7 +121,7 @@ Present as a prioritized list — no AUQ needed, just a summary the user can act Update the release tracking issue checkboxes: 1. Run `gh issue view {number} --json body` and capture the output. Parse the JSON to extract the `body` field, then write it to `/tmp/release-issue-body.md` using the Write tool (not shell redirection). -2. Read the file with the Read tool. Each `/release`-managed checkbox in the template has an HTML comment label like ``, ``, or ``. To mark a step complete, find the line containing the matching label and change its `- [ ]` to `- [x]`. Use the Edit tool to write the updated body back to the file. In Phase 2 Release readiness, mark `release:core` complete. +2. Read the file with the Read tool. Each `/release`-managed checkbox in the template has an HTML comment label like ``, ``, ``, or ``. To mark a step complete, find the line containing the matching label and change its `- [ ]` to `- [x]`. Use the Edit tool to write the updated body back to the file. In Phase 2 Release readiness, mark `release:core` complete. 3. Run `gh issue edit {number} --body-file /tmp/release-issue-body.md` to push the updates. Then present a summary and next action via AskUserQuestion: @@ -148,14 +148,33 @@ pwsh -Command "./src/scripts/Package-Toolkit.ps1 -Build -CopyFiles" If it fails, show the error and ask whether to investigate or skip. +### Package Power BI + +Power BI packaging reports its own state and saves demo reports with Power BI Desktop, which only runs on Windows. + +On Windows, tell the user not to use the mouse or keyboard until it finishes, then run: + +```bash +pwsh -Command "./src/scripts/Package-PowerBI.ps1 -Unattended" +``` + +On other platforms, run `pwsh -Command "./src/scripts/Package-PowerBI.ps1 -Status"` and tell the user to run `Package-PowerBI.ps1 -Unattended` on a Windows machine with Power BI Desktop, then say "done". + +It takes about an hour (roughly 10 minutes per demo report). If it stops because the reports were built from a test open data URL ("isn't permanent"), rerun it once with `-Build` to rebuild against the main branch; don't use `-Build` otherwise, because it discards reports that are already saved. + +Relay its status output verbatim — it says exactly which step is next. If a report can't be saved automatically, relay the failed step and screenshot path, tell the user to run `Package-PowerBI.ps1 -Open` and save that project as PBIX (refresh, Save as PBIX in `release/pbix`, sensitivity "Public"), then say "done" so you can rerun the command to validate and package. + +If validation fails, show the reported issues verbatim. Each one names the file and the fix; don't guess at causes. + +Do not proceed until all three files exist: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. + ### Manual steps reminder After packaging succeeds, inform the user of remaining manual steps documented in the release checklist issue (include a link): -1. Power BI packaging -2. Publish release -3. Publish announcements +1. Publish release +2. Publish announcements ### Final issue update -Update the release tracking issue checkboxes for finalize-phase items using the same label-based process as Release readiness. Mark `release:finalize` and `release:package` complete. Then report completion and wish the user well. +Update the release tracking issue checkboxes for finalize-phase items using the same label-based process as Release readiness. Mark `release:finalize` and `release:package` complete. Mark `release:powerbi` complete only when all three Power BI files exist. Then report completion and wish the user well. diff --git a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md index 75c7f3599..1ee3e6a61 100644 --- a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md +++ b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md @@ -110,15 +110,18 @@ Status icons: > _This step is optional, but can catch issues earlier. You can also add the `-Build` parameter to the publish command in the next step._ - Ensure all tests pass: `/src/scripts/Test-PowerShell -Unit -Integration` - [ ] Package all release files (except Power BI): `/src/scripts/Package-Toolkit.ps1 -Build -CopyFiles` script -- [ ] Package Power BI files - - [ ] Run `Package-Toolkit.ps1 -OpenPBI` script. - - [ ] Save and close each Power BI project: - - Select the `/release/pbix` folder. - - Change the file extension to PBIX. - - When prompted, set the sensitivity to "Public". - - Manually remove unused queries based on what's documented in Build-PowerBI.ps1 ~line 120. - - Verify all pages, switch to the Get started page, and save again. - - [ ] Run `Package-Toolkit -ZipPBI` script. +- [ ] Confirm the data source saved in the Power BI semantic models points at the current demo hub. +- [ ] Confirm the demo hub is on this release and has current data: `/src/scripts/Deploy-Demo.ps1 -Check` + - If it isn't, update it with `Deploy-Demo.ps1` and wait for ingestion before packaging Power BI. +- [ ] Package Power BI files on Windows: `/src/scripts/Package-PowerBI.ps1 -Unattended` + > _Builds, saves demo PBIX files with Power BI Desktop, validates, and packages. Takes about an hour. Don't use the mouse or keyboard while it runs. If it stops because the reports were built from a test open data URL, rerun once with `-Build`._ + > + > _First, sign in to Power BI Desktop and refresh one demo project by hand so the data source credentials are stored and the "Public" sensitivity label is available. The command can't sign in or answer a credential prompt._ + - If a report can't be saved automatically, the command names the step that failed and saves a screenshot. Save that report by hand, then rerun the command: + - Run `Package-PowerBI.ps1 -Open` to open the projects that need to be saved. + - Refresh the report, select **File** > **Save as**, keep the `/release/pbix` folder, change the file type to PBIX, and set the sensitivity to "Public". + > _Queries are already trimmed and the Get started page is already selected. There is nothing else to change._ + - [ ] Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. - [ ] Check the docs for broken links: - Create a personal fork of the main repo. - If you already have one, update it to the latest. diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 1702ee02d..7fd648f3b 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -11,6 +11,12 @@ on: - 'src/templates/finops-hub/**/timeZones.bicep' - 'docs/deploy/finops-hub-latest.json' - 'docs/deploy/finops-hub-preview.json' + # Power BI report definitions and build scripts are covered by PowerBIReports.Tests.ps1 and Build-PowerBI.Tests.ps1 + - 'src/power-bi/kql/**' + - 'src/power-bi/storage/**' + - 'src/power-bi/reports.json' + - 'src/scripts/Build-PowerBI.ps1' + - 'src/scripts/Package-PowerBI.ps1' jobs: run_pester_tests: name: Pester diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 6ca4dd7c3..ad350f208 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 09/11/2026 +ms.date: 10/14/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -133,8 +133,13 @@ _Released June 2026_ - **Changed** - Updated report labels and documentation to use current FinOps Framework capability names, including Usage optimization and Governance, policy, and risk ([#2170](https://github.com/microsoft/finops-toolkit/pull/2170)). - **Fixed** + - Fixed report templates opening with a preconfigured demo data source. Storage and cluster settings are now empty when you open a template. + - Fixed storage report refresh errors for exports using earlier FOCUS versions, which don't have the `CapacityReservationId` and `CapacityReservationStatus` columns. They're now added as empty columns, like `AvailabilityZone` already was. + - Fixed the managed disks table on the **Virtual machines** page of the [Governance](power-bi/governance.md) report, which showed an error because the report template didn't include the disks data. - Fixed Power BI storage report refresh errors caused by ISO 8601 duration `x_SkuTerm` values (like `P3Y`) and empty strings in cost exports ([#2174](https://github.com/microsoft/finops-toolkit/issues/2174)). - Paginate Azure Resource Graph queries by subscription to mitigate [payload size limit](help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) errors in the [Governance](power-bi/governance.md) and [Workload optimization](power-bi/workload-optimization.md) reports ([#1768](https://github.com/microsoft/finops-toolkit/issues/1768)). Queries still surface a payload size error (rather than silently returning truncated results) if a single batch of subscriptions exceeds the limit; see [Reduce the batch size](help/errors.md#option-1-reduce-the-batch-size) for the mitigation. As part of this change, the `AdvisorRecommendations` and `AdvisorReservationRecommendations` tables now return complete (non-truncated) results per batch, matching every other batched table; previously these two tables silently dropped rows beyond the payload limit instead of erroring. + - Fixed the [Governance](power-bi/governance.md) report template missing the policy definitions query. + - Fixed report templates showing tables in the model diagram that the report doesn't use. ### [FinOps workbooks](workbooks/finops-workbooks-overview.md) v15 diff --git a/docs-wiki/Build-and-test.md b/docs-wiki/Build-and-test.md index 8410afcd9..c151ef9b3 100644 --- a/docs-wiki/Build-and-test.md +++ b/docs-wiki/Build-and-test.md @@ -132,16 +132,23 @@ The Build-PowerShell script calls an Invoke-Build task internally. ### Building Power BI reports -Power BI reports cannot be built automatically. Generating PBIT and PBIX files must be done manually: - -1. Open the desired Power BI report. -2. Save the report as a `.pbix` file using the same name in the `releases` folder. -3. Remove unnecessary queries. -4. Remove setup instructions for unnecessary parameters. -5. Remove unnecessary parameters. -6. Save the report again. -7. Copy the first paragraph description from the main page. -8. Save the report as a `.pbit` file using the copied description and add "To learn more, see https://aka.ms/ftk/". +Run [Build-PowerBI](../src/scripts/README.md#-build-powerbi) to generate the Power BI release artifacts: + +```powershell +cd /src/scripts +./Build-PowerBI +``` + +This generates two things from a single prune, so the template and the demo report can't drift: + +- PBIT templates in `release/pbit`, zipped into `PowerBI-kql.zip` and `PowerBI-storage.zip`. +- PBIP projects for demo reports in `release/pbix`, each containing only the tables, relationships, and queries that report needs. + +The tables and queries each report keeps are listed in [src/power-bi/reports.json](../src/power-bi/reports.json). When a report starts using a new table or query, add it there. The build and the `PowerBIReports` lint test fail when a visual or query uses something the report doesn't keep. + +Demo projects keep the data source saved in the semantic model. They read open data from the `main` branch, because the release URL resolves to the last published release and doesn't have open data files added during the release being built. Templates always ship with the data source set to null and the release open data URL. Only permanent URLs are accepted, because the URL ships inside the demo reports. To test before the merge to `main`, add `-TestOpenDataUrl` and pass a branch with `-OpenDataUrl`. Reports built that way are not packaged. + +Demo PBIX files need Power BI Desktop to load and save them. On Windows, run [Package-PowerBI](../src/scripts/README.md#-package-powerbi) `-Unattended` to save, validate, and package them automatically. ### Building documentation diff --git a/docs-wiki/Release-process.md b/docs-wiki/Release-process.md index 55deb704f..0d50fd3e0 100644 --- a/docs-wiki/Release-process.md +++ b/docs-wiki/Release-process.md @@ -121,21 +121,28 @@ Once the above requirements have been met, the feature branch can be merged into > _The documentation site may take 5 minutes to update after the merge is committed. If not updated, look at [GitHub actions](../actions/workflows/pages/pages-build-deployment) to see if there are any failures._ - 6. Run `Package-Toolkit -Build -PowerBI` script. - - For each Power BI report: - 1. Save the file as a PBIX file to the release folder. - 2. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠️ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - 3. Update the version on the **Get started** tab. - 4. For the Cost summary and Data ingestion reports, remove the following from the Transform data (query editor): - 1. Delete both **Recommendations\*** queries. - 2. Delete the **InstanceSizeFlexibility** query. - 3. Open the **▶️ START HERE** query in the advanced editor and remove connector settings and generated rows in the table from the separator line to the end. - 5. For the Cost summary and Rate optimization reports, remove the following from the Transform data (query editor): - 1. Delete all **Hub\*** queries. - 6. Save PBIX again in the release folder. - > ⚠️ _**DO NOT** save the above changes back to the Power BI project files!_ - 7. Copy the first paragraph from the **Get started** page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/{report-name}" as a separate paragraph in the description. + 6. Package the Power BI files on Windows with Power BI Desktop installed: + + ```powershell + cd /src/scripts + ./Package-PowerBI -Unattended + ``` + + > _Builds the templates, saves each demo report as a PBIX file with Power BI Desktop, validates the files, and packages them. Takes about an hour. Don't use the mouse or keyboard while it runs. If it stops because the reports were built from a test open data URL, rerun once with `-Build`._ + > + > _First, sign in to Power BI Desktop and refresh one demo project from `/release/pbix` by hand. That stores the data source credentials and makes the **Public** sensitivity label available. The command can't sign in or answer a credential prompt: a sign-in window stops it right away, and missing credentials fail the refresh._ + > + > ⚠️ _If the sensitivity label isn't **Public**, the report will not open for anyone outside of Microsoft. Validation fails when a demo report has any other label._ + + If a report can't be saved automatically, the command names the step that failed and saves a screenshot next to the PBIX file. Save that report by hand, then rerun the command: + + 1. Run `./Package-PowerBI -Open` to open the projects that still need to be saved. + 2. Refresh the report so demo data is loaded. + 3. Select **File** > **Save as**, keep the `/release/pbix` folder, and change the file type to PBIX. + 4. Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. + > _Queries are already trimmed and the **Get started** page is already selected. There is nothing else to change._ + + Confirm all 3 files were created: PowerBI-kql.zip, PowerBI-storage.zip, and PowerBI-demo.zip. 7. Tag and publish a [new release](../releases/new): 1. Create a tag on publish using the "vX.X" format. 2. Set the **Target** to `main`. diff --git a/src/power-bi/README.md b/src/power-bi/README.md index 9cbc3ca4c..f521a01f9 100644 --- a/src/power-bi/README.md +++ b/src/power-bi/README.md @@ -17,7 +17,7 @@ Power BI reports use [Tabular Model Definition Language (TMDL)](https://learn.mi 3. Enable the following features: 1. Power BI Project (.pbip) save option 2. Store semantic model using TMDL format -4. **DO NOT** enable the "Store reports using enhanced metadata format (PBIR)" +4. The "Store reports using enhanced metadata format (PBIR)" setting no longer applies. PBIR has been generally available and the default report format since September 2026, and Power BI Desktop converts reports to it when saving. Power BI Desktop keeps a backup of the previous format for 30 days under `%USERPROFILE%\Microsoft\Power BI Desktop Store App\TempSaves\Backups`. 5. Select the **OK** button to save settings 6. Restart Power BI @@ -65,6 +65,11 @@ Power BI reports in the FinOps toolkit must adhere to the following guidelines: 2. Queries should all be a the root **FinOps toolkit** folder. 1. All queries should use the same settings. Do not duplicate properties that can be shared. 7. All text (including report, page, and column names) should follow the [content guidelines](../../docs-wiki/Coding-guidelines.md#-content-strings-and-microcopy). +8. The data source saved in the semantic model is the demo hub, which is what demo reports connect to. Update it when the demo hub changes. Templates always ship with it set to null. +9. Every table and query a report uses must be listed for that report in [reports.json](./reports.json). + 1. Reports share one semantic model per data source. The release build removes everything a report doesn't list, so an unlisted table shows up as a broken visual or refresh error. + 2. Prefix a name with `[kql]` or `[storage]` when only one report type uses it. + 3. The `PowerBIReports` lint test fails when a visual uses a table or column the report doesn't list, or a list entry names something that doesn't exist. It doesn't check which tables a kept query or measure depends on; that check runs in `Build-PowerBI` at release time, so list everything a query references. > [!NOTE] > Remember: These are general rules of thumb for consistency. They are not set in stone. Please propose updates by submitting a PR, starting a discussion, or creating an issue. In general, use your best judgement to make the report as useful as possible. We just want everything to feel consistent and familiar. diff --git a/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl b/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl index 2dab7934a..5de53c619 100644 --- a/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl +++ b/src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl @@ -380,14 +380,6 @@ table AdvisorReservationRecommendations annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.Region' - dataType: string - lineageTag: 68593266-9950-4caf-9522-ccfcbaa5d5e3 - summarizeBy: none - sourceColumn: properties.extendedProperties.Region - - annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.commitment' dataType: string lineageTag: e0b87d0c-8c42-4354-b85a-5fc93d4af06a @@ -469,11 +461,10 @@ table AdvisorReservationRecommendations ['properties.extendedProperties.DatabaseSize'] = properties.extendedProperties.DatabaseSize, ['properties.extendedProperties.DatabaseName'] = properties.extendedProperties.DatabaseName, ['properties.extendedProperties.ServerName'] = properties.extendedProperties.ServerName, - ['properties.extendedProperties.Region'] = properties.extendedProperties.Region, ['properties.extendedProperties.displaySKU'] = properties.extendedProperties.displaySKU, ['properties.extendedProperties.displayQty'] = toint(properties.extendedProperties.displayQty), ['properties.extendedProperties.location'] = properties.extendedProperties.location, - ['properties.extendedProperties.region'] = properties.extendedProperties.region, + ['properties.extendedProperties.region'] = coalesce(tostring(properties.extendedProperties.region), tostring(properties.extendedProperties.Region)), ['properties.extendedProperties.vmSize'] = properties.extendedProperties.vmSize, ['properties.extendedProperties.subId'] = properties.extendedProperties.subId, ['properties.extendedProperties.scope'] = properties.extendedProperties.scope, @@ -496,7 +487,7 @@ table AdvisorReservationRecommendations NullHandling = if Source = null then #table( - { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, + { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, {} ) else Source, diff --git a/src/power-bi/reports.json b/src/power-bi/reports.json new file mode 100644 index 000000000..de294f101 --- /dev/null +++ b/src/power-bi/reports.json @@ -0,0 +1,172 @@ +{ + "$comment": [ + "Build settings for each Power BI report, used by src/scripts/Build-PowerBI.ps1.", + "description: Template description shown when the PBIT file is opened.", + "tables: Tables to keep. Everything else is removed from the template and demo project.", + "queries: Parameters, functions, and queries that do not load a table (Power Query expressions) to keep.", + "Prefix a name with [kql] or [storage] to keep it for that report type only." + ], + "reports": { + "CostSummary": { + "description": "The Cost summary report provides several summaries of your effective (amortized) and billed costs based on the FinOps Open Cost and Usage Specification (FOCUS). Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit. Effective costs will not match your invoice.", + "tables": ["Costs", "Prices", "PricingUnits"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "DataIngestion": { + "description": "The Data ingestion report provides details about the data you've ingested into your FinOps hub storage account.", + "tables": ["Costs", "HubScopes", "HubSettings", "Prices", "PricingUnits", "StorageData", "StorageErrors"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "Governance": { + "description": "The Governance, policy, and risk report summarizes your Microsoft Cloud governance posture. It offers the standard metrics aligned with the Cloud Adoption Framework to facilitate identifying issues, applying recommendations, and resolving compliance gaps.", + "tables": [ + "AdvisorRecommendations", + "Compliance calculation", + "Costs", + "Disks", + "ManagementGroups", + "NetworkInterfaces", + "NetworkSecurityGroups", + "PolicyAssignments", + "PolicyStates", + "Prices", + "PricingUnits", + "PublicIPAddresses", + "Regions", + "Resources", + "ResourceTypes", + "SqlDatabases", + "Subscriptions", + "VirtualMachines", + "VirtualMachinesDisks" + ], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "PolicyDefinitions", + "Remove Duplicate Resource IDs", + "Deprecated: Perform Extra Query Optimizations", + "ftk_ARGBatchSize", + "ftk_DemoFilter", + "ftk_QueryARG", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "Invoicing": { + "description": "The Invoicing and chargeback report provides several summaries of your billed cost to facilitate invoice reconciliation for Microsoft Customer Agreement (MCA) and Enterprise Agreement (EA) accounts or to perform chargeback using effective (amortized) costs.", + "tables": ["Costs", "Prices", "PricingUnits"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "RateOptimization": { + "description": "The Rate optimization report provides insights into any rate optimization opportunities, like reservations, savings plans, and Azure Hybrid Benefit. This report uses effective cost, which amortizes and breaks reservation and savings plan purchases down and allocates costs out to the resources that received the benefit. Effective cost will not match your invoice.", + "tables": ["Costs", "InstanceSizeFlexibility", "Prices", "PricingUnits", "ReservationRecommendations"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Deprecated: Perform Extra Query Optimizations", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + }, + "WorkloadOptimization": { + "description": "The Usage optimization report provides insights into resource utilization and efficiency opportunities based on historical usage patterns. Use this report to determine if resources can be scaled down or even shut down during off-peak hours to minimize wasteful usage and spending. Also consider cheaper alternatives when available and ensure all workloads have some direct or indirect link to business value to avoid unnecessary usage and costs that don't contribute to the mission.", + "tables": ["AdvisorRecommendations", "Costs", "Disks", "Prices", "PricingUnits", "Resources", "Subscriptions"], + "queries": [ + "▶️ START HERE", + "Cluster URL", + "[storage]Storage URL", + "Default Granularity", + "Number of Months", + "RangeStart", + "RangeEnd", + "Experimental: Add Missing Prices", + "Remove Duplicate Resource IDs", + "Deprecated: Perform Extra Query Optimizations", + "ftk_ARGBatchSize", + "ftk_DemoFilter", + "ftk_QueryARG", + "ftk_DatetimeToJulianDate", + "ftk_ImpalaToJulianDate", + "ftk_Metadata", + "ftk_ParseResourceId", + "ftk_ParseResourceName", + "ftk_ParseResourceType", + "ftk_Storage" + ] + } + } +} diff --git a/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl b/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl index 047210e6a..0d85eb1e3 100644 --- a/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl +++ b/src/power-bi/storage/Shared.Dataset/definition/relationships.tmdl @@ -64,10 +64,6 @@ relationship 2ad9b8bc-59fb-e94c-a36f-1fa66aaa02a8 fromColumn: AdvisorRecommendations.'properties.resourceMetadata.resourceId' toColumn: Resources.id -relationship AutoDetected_d442ee36-a039-47be-83cf-fd058a0973d7 - fromColumn: AdvisorReservationRecommendations.subscriptionId - toColumn: Subscriptions.subscriptionId - relationship AutoDetected_19cbab3d-1150-45a4-9258-ca0b47ba687d isActive: false fromColumn: AdvisorRecommendations.subscriptionId diff --git a/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl b/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl index 2dab7934a..5de53c619 100644 --- a/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl +++ b/src/power-bi/storage/Shared.Dataset/definition/tables/AdvisorReservationRecommendations.tmdl @@ -380,14 +380,6 @@ table AdvisorReservationRecommendations annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.Region' - dataType: string - lineageTag: 68593266-9950-4caf-9522-ccfcbaa5d5e3 - summarizeBy: none - sourceColumn: properties.extendedProperties.Region - - annotation SummarizationSetBy = Automatic - column 'properties.extendedProperties.commitment' dataType: string lineageTag: e0b87d0c-8c42-4354-b85a-5fc93d4af06a @@ -469,11 +461,10 @@ table AdvisorReservationRecommendations ['properties.extendedProperties.DatabaseSize'] = properties.extendedProperties.DatabaseSize, ['properties.extendedProperties.DatabaseName'] = properties.extendedProperties.DatabaseName, ['properties.extendedProperties.ServerName'] = properties.extendedProperties.ServerName, - ['properties.extendedProperties.Region'] = properties.extendedProperties.Region, ['properties.extendedProperties.displaySKU'] = properties.extendedProperties.displaySKU, ['properties.extendedProperties.displayQty'] = toint(properties.extendedProperties.displayQty), ['properties.extendedProperties.location'] = properties.extendedProperties.location, - ['properties.extendedProperties.region'] = properties.extendedProperties.region, + ['properties.extendedProperties.region'] = coalesce(tostring(properties.extendedProperties.region), tostring(properties.extendedProperties.Region)), ['properties.extendedProperties.vmSize'] = properties.extendedProperties.vmSize, ['properties.extendedProperties.subId'] = properties.extendedProperties.subId, ['properties.extendedProperties.scope'] = properties.extendedProperties.scope, @@ -496,7 +487,7 @@ table AdvisorReservationRecommendations NullHandling = if Source = null then #table( - { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, + { "id", "name", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.reservedResourceType", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.targetResourceCount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.displaySKU", "properties.extendedProperties.displayQty", "properties.extendedProperties.location", "properties.extendedProperties.region", "properties.extendedProperties.vmSize", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.qty", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "recommendedActions", "SortOrder" }, {} ) else Source, diff --git a/src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl b/src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl index 75880048f..be4922bdd 100644 --- a/src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl +++ b/src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl @@ -1480,8 +1480,8 @@ table Costs // Fix known bugs and set data types Fixes = Table.TransformColumnTypes( Table.ReplaceValue(Table.ReplaceValue( - // Create AvailabilityZone column if missing to support non-Microsoft clouds - if Table.HasColumns(Align12, {"AvailabilityZone"}) then Align12 else Table.AddColumn(Align12, "AvailabilityZone", each null), + // Create columns the report needs that older FOCUS versions and non-Microsoft clouds don't have + List.Accumulate({"AvailabilityZone", "CapacityReservationId", "CapacityReservationStatus"}, Align12, (tbl, col) => if Table.HasColumns(tbl, {col}) then tbl else Table.AddColumn(tbl, col, each null)), // Fix savings plan purchases -- Cost Management fix pending as of Dec 8, 2024 each [EffectiveCost], each if [ChargeCategory] = "Purchase" and [CommitmentDiscountId] <> "" and [CommitmentDiscountId] <> null then 0.0 else [EffectiveCost], Replacer.ReplaceValue, {"EffectiveCost"}), // Fix PricingCategory values -- Remove after Oct 1, 2025 diff --git a/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 b/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 new file mode 100644 index 000000000..50330e364 --- /dev/null +++ b/src/powershell/Tests/Lint/PowerBIReports.Tests.ps1 @@ -0,0 +1,83 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + Lint rule: Power BI reports and src/power-bi/reports.json must agree. + + Build-PowerBI removes every table and query that reports.json doesn't list for a report. When a + visual reads from a table that isn't listed, the template and demo report ship with a broken + visual (for example, the Governance report's "Virtual machines" page lost its disks table this + way). Build-PowerBI fails on this too, but it only runs at release time. This test catches it + in the pull request that causes it. + + Uses the same helpers as Build-PowerBI and only reads files, so it doesn't need the Analysis + Services library. +#> + +BeforeDiscovery { + $powerBiDir = Join-Path (Get-Item -Path $PSScriptRoot).Parent.Parent.Parent.FullName 'power-bi' + $reportProjects = @(Get-ChildItem $powerBiDir -Recurse -Include '*.kql.pbip', '*.storage.pbip' | ForEach-Object { + @{ Name = $_.Name.Split('.')[0]; Type = $_.Name.Split('.')[1]; Folder = $_.DirectoryName; Label = $_.BaseName } + }) +} + +BeforeAll { + $srcDir = (Get-Item -Path $PSScriptRoot).Parent.Parent.Parent.FullName + $powerBiDir = Join-Path $srcDir 'power-bi' + + $ast = [System.Management.Automation.Language.Parser]::ParseFile((Join-Path $srcDir 'scripts/Build-PowerBI.ps1'), [ref]$null, [ref]$null) + $ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] }, $false) ` + | ForEach-Object { . ([scriptblock]::Create($_.Extent.Text)) } + + $config = (Get-Content (Join-Path $powerBiDir 'reports.json') -Raw | ConvertFrom-Json -Depth 10).reports + $modelNames = @{} + foreach ($type in 'kql', 'storage') + { + $modelNames[$type] = ConvertTo-NameSet (Get-TmdlObjectName (Join-Path $powerBiDir "$type/Shared.Dataset/definition")) + } +} + +Describe 'PowerBIReports' { + Context '