diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 6ca4dd7c3..0b49f881b 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 09/11/2026 +ms.date: 10/14/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -34,6 +34,7 @@ The following section lists features and enhancements that are currently in deve - Replaced redundant `tolower()` comparisons in hub KQL with case-insensitive operators (`has`, `=~`, `!~`) so the engine can use the term index instead of scanning every row ([#2213](https://github.com/microsoft/finops-toolkit/issues/2213)). - Replaced whole-term `contains` matches with `has` across hub KQL and the query catalog (resource ID paths, licensing phrases, SKU description terms) and added a per-row operator-equivalence regression harness with unit test coverage ([#2220](https://github.com/microsoft/finops-toolkit/pull/2220)). - **Fixed** + - Fixed hub redeploys silently resetting the stored ingestion and Data Explorer final retention to the 13-month default when a custom value wasn't explicitly re-specified, which caused the purge pipeline to age out historical data on the next run. Stored values are no longer changed by a redeploy; edit `config/settings.json` to change them ([#2206](https://github.com/microsoft/finops-toolkit/issues/2206)). - Fixed ADF schedule trigger deployment failures (`InvalidWorkflowTriggerRecurrence`) in regions not covered by the region-to-time-zone map (for example, Sweden Central), where the trigger's `startTime` was missing the required `Z` suffix for the `UTC` fallback time zone ([#2157](https://github.com/microsoft/finops-toolkit/issues/2157), [#2291](https://github.com/microsoft/finops-toolkit/pull/2291)). - Fixed managed exports failing with an `Unauthorized` error because the Role Based Access Control Administrator role was never assigned to the Data Factory identity. Roles requested by a hub app are now assigned on the publisher storage account even when the app doesn't create the storage account itself ([#2253](https://github.com/microsoft/finops-toolkit/issues/2253)). - Fixed private-network deployments that Azure Policy blocked when `defaultOutboundAccess` was omitted. Private mode subnets now set it to `false`, while an Azure Files private endpoint supports deployment-script storage and the NAT Gateway provides required container egress ([#2258](https://github.com/microsoft/finops-toolkit/issues/2258), [#2259](https://github.com/microsoft/finops-toolkit/pull/2259)). diff --git a/docs-mslearn/toolkit/hubs/data-processing.md b/docs-mslearn/toolkit/hubs/data-processing.md index 9c3e1cabb..86d9ed908 100644 --- a/docs-mslearn/toolkit/hubs/data-processing.md +++ b/docs-mslearn/toolkit/hubs/data-processing.md @@ -3,7 +3,7 @@ title: FinOps hubs data processing description: Learn how FinOps hubs process data, including scope setup, data normalization, and optimization, to enhance cost management and analysis. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 10/14/2026 ms.topic: concept-article ms.service: finops ms.subservice: finops-toolkit @@ -53,7 +53,7 @@ The following diagram depicts the end-to-end data ingestion process within FinOp 3. When data is ingested into raw tables in Data Explorer, an update policy copies the data into the corresponding `{dataset}_final_v1_0` table using the `{dataset}_transform_v1_0()` function to normalize all data to align to FOCUS 1.0. 4. After ingestion, the **ingestion_ETL_dataExplorer** pipeline performs some cleanup, including purging data in the final table that is past the data retention period. - As of 0.7, Data Explorer applies data retention in raw tables while data retention in final tables is applied by the ingestion pipeline. If data ingestion stops, historical data isn't purged. - - Data retention can be configured during the template deployment or manually in the **config/settings.json** file in storage. + - Data retention can be configured during the template deployment or manually in the **config/settings.json** file in storage. Once a hub has stored ingestion or final retention values, redeploying with a different `ingestionRetentionInMonths` or `dataExplorerFinalRetentionInMonths` doesn't change them, so a lower value never purges data by accident. To change retention later, edit **config/settings.json** in the hub storage account. 6. (Optional) The **queries_DailySchedule** trigger runs the **queries_ExecuteETL** pipeline once per day to query Azure Resource Graph for additional recommendations and saves results to the **ingestion/Recommendations** folder. [Learn more](#azure-resource-graph-recommendations). 7. Reports and other tools like Power BI read data from Data Explorer or the **ingestion** container. - Data in Data Explorer can be read from the **Hub** database. diff --git a/src/powershell/Tests/Unit/HubsRetentionGuard.Tests.ps1 b/src/powershell/Tests/Unit/HubsRetentionGuard.Tests.ps1 new file mode 100644 index 000000000..04c59a5d4 --- /dev/null +++ b/src/powershell/Tests/Unit/HubsRetentionGuard.Tests.ps1 @@ -0,0 +1,92 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + Regression coverage for the settings.json retention guard (#2206): + Copy-FileToAzureBlob.ps1 unconditionally overwrote retention.ingestion.months / retention.final.months + with whatever the deploymentScript's Bicep parameters passed in. Bicep always resolves a value for an + optional parameter (defaulting to 13 if the caller didn't specify one), so the script cannot tell an + explicit redeploy value from a silently-defaulted one. A redeploy that omitted a previously-customized + retention value therefore silently reset it to the toolkit default, and the next purge pipeline run + aged out historical data older than the new (lower) cutoff -- oldest data first. + + The fix: a value already stored in settings.json is kept on redeploy. The requested value only applies + when nothing is stored yet (new hub, or settings.json from before retention existed). + + These tests run the real script with stubbed Az cmdlets and assert on the JSON it uploads. +#> + +Describe 'HubsRetentionGuard' { + + BeforeAll { + $repoRoot = (Resolve-Path "$PSScriptRoot/../../../..").Path + $scriptPath = Join-Path $repoRoot 'src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1' + + function Invoke-CopyScript + { + param([string] $ExistingJson, [int] $Ingestion, [int] $Final) + + $workDir = Join-Path ([IO.Path]::GetTempPath()) ([guid]::NewGuid().ToString()) + New-Item -ItemType Directory -Path $workDir | Out-Null + $existingJson = $ExistingJson + + function New-AzStorageContext { param($StorageAccountName, [switch] $UseConnectedAccount) 'ctx' } + function Get-AzStorageBlobContent + { + param($Context, $Container, $Blob, $Destination, [switch] $Force) + if ($null -eq $existingJson) { return $null } + Set-Content -Path $Destination -Value $existingJson + return 'blob' + } + function Set-AzStorageBlobContent { param($Context, $Container, $File, [switch] $Force) } + + $env:storageAccountName = 'sa' + $env:containerName = 'config' + $env:scopes = '' + $env:ftkVersion = '0.0.0' + $env:msexportRetentionInDays = '0' + $env:rawRetentionInDays = '0' + $env:ingestionRetentionInMonths = "$Ingestion" + $env:finalRetentionInMonths = "$Final" + + Push-Location $workDir + try + { + & $scriptPath *> $null + # The script writes the final settings.json to the working directory before uploading it + return (Get-Content -Path (Join-Path $workDir settings.json) -Raw | ConvertFrom-Json) + } + finally { Pop-Location; Remove-Item $workDir -Recurse -Force -ErrorAction SilentlyContinue } + } + } + + Context 'Existing settings.json' { + + It 'Should keep a stored value that is higher than the requested one' { + $r = Invoke-CopyScript -ExistingJson '{"version":"","scopes":[],"retention":{"msexports":{"days":0},"ingestion":{"months":36},"raw":{"days":0},"final":{"months":60}}}' -Ingestion 13 -Final 13 + $r.retention.ingestion.months | Should -Be 36 -Because 'a redeploy without an explicit value must not reset customized retention (#2206)' + $r.retention.final.months | Should -Be 60 + } + + It 'Should keep a stored value that is lower than the requested one' { + $r = Invoke-CopyScript -ExistingJson '{"version":"","scopes":[],"retention":{"msexports":{"days":0},"ingestion":{"months":6},"raw":{"days":0},"final":{"months":6}}}' -Ingestion 13 -Final 13 + $r.retention.ingestion.months | Should -Be 6 -Because 'a stored value is never changed on redeploy' + $r.retention.final.months | Should -Be 6 + } + + It 'Should seed retention from the request when settings.json predates retention' { + $r = Invoke-CopyScript -ExistingJson '{"version":"","scopes":[]}' -Ingestion 24 -Final 30 + $r.retention.ingestion.months | Should -Be 24 + $r.retention.final.months | Should -Be 30 + } + } + + Context 'First deploy' { + + It 'Should honor the requested retention when no settings.json exists' { + $r = Invoke-CopyScript -ExistingJson $null -Ingestion 6 -Final 8 + $r.retention.ingestion.months | Should -Be 6 -Because 'there is no stored value to protect on a brand-new hub' + $r.retention.final.months | Should -Be 8 + } + } +} diff --git a/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1 b/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1 index ffea9d454..05bd1b9d5 100644 --- a/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1 +++ b/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1 @@ -59,6 +59,15 @@ if ($blob) } } +# Remember retention values already stored in settings.json before any defaults are seeded below +$storedIngestionMonths = $null +$storedFinalMonths = $null +if ($json -and $json.retention) +{ + if ($json.retention.ingestion -and $null -ne $json.retention.ingestion.months) { $storedIngestionMonths = [Int32]$json.retention.ingestion.months } + if ($json.retention.final -and $null -ne $json.retention.final.months) { $storedFinalMonths = [Int32]$json.retention.final.months } +} + # Set default if not found if (!$json) { @@ -124,14 +133,18 @@ else $json.retention.msexports.days = [Int32]::Parse($env:msexportRetentionInDays) } -# Set or update ingestion retention +# Set ingestion retention +# NOTE: Bicep always passes a value here (defaulting to 13 if the caller didn't specify one), so this script +# cannot tell an explicit redeploy value from a silently-defaulted one. A value already stored in settings.json +# is therefore never changed on redeploy -- a silent change ages out data the purge pipeline acts on (#2206). +# The requested value only applies when nothing was stored yet (new hub or pre-0.4 settings). if (!($json.retention.ingestion)) { $json.retention | Add-Member -Name ingestion -Value (ConvertFrom-Json "{""months"":$($env:ingestionRetentionInMonths)}") -MemberType NoteProperty } else { - $json.retention.ingestion.months = [Int32]::Parse($env:ingestionRetentionInMonths) + $json.retention.ingestion.months = if ($null -ne $storedIngestionMonths) { $storedIngestionMonths } else { [Int32]::Parse($env:ingestionRetentionInMonths) } } # Set or update raw retention @@ -144,14 +157,14 @@ else $json.retention.raw.days = [Int32]::Parse($env:rawRetentionInDays) } -# Set or update final retention +# Set final retention (keeps any stored value -- see note above) if (!($json.retention.final)) { $json.retention | Add-Member -Name final -Value (ConvertFrom-Json "{""months"":$($env:finalRetentionInMonths)}") -MemberType NoteProperty } else { - $json.retention.final.months = [Int32]::Parse($env:finalRetentionInMonths) + $json.retention.final.months = if ($null -ne $storedFinalMonths) { $storedFinalMonths } else { [Int32]::Parse($env:finalRetentionInMonths) } } # Updating settings