fix(hubs): paginate Azure Resource Graph queries past 1,000 rows

[Michael Flanakin (flanakin)]

🤖 [AI]

Summary

• Azure Resource Graph caps a single query response at 1,000 rows and returns a $skipToken when more results exist, but the queries_ResourceGraph_ExecuteQuery pipeline ran a single Copy activity with no pagination, silently truncating larger result sets (for example Recommendations-Microsoft-AdvisorCost).
• The ADF REST connector's paginationRules can only inject a continuation value into a URL, query string, or header — not a POST body — so it can't express ARG's $skipToken continuation on its own.
• Replaced the single Copy activity with an Until loop: each iteration runs the query via a Web activity to check for results and read $skipToken, then re-runs the same page via Copy (to keep schema mapping/translator support) and writes it to its own _{page}.parquet file. The loop stops once ARG stops returning a token.

Test plan

• bicep build src/templates/finops-hub/main.bicep --stdout compiles cleanly
• Invoke-Pester -Path ./src/powershell/Tests/Unit/HubsIngestionQueries.Tests.ps1 — 192/192 passed
• Manual: deploy a hub, run queries_ExecuteETL against a tenant with >1,000 Advisor cost recommendations, confirm multiple _{page}.parquet files land in ingestion storage and all rows reach the destination table

Fixes #2320

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

Reviewed the pagination rework end to end. The Until shape is right and the diagnosis matches what I see in the module.

Things I checked that are fine, so you don't have to re-derive them: the _{page} suffix doesn't break ADX table routing (the table comes from the folder path, not the file name) or the drop-extents cleanup (per-file drop-by: tags plus a per-run ingestionId means no stale pages and no upgrade hazard); manifest gating means a mid-loop failure can't half-ingest; and the request-body JSON escaping and the zero-row exit both behave the same as the old | count path.

Five things below. The first two are the ones I'd want resolved before this merges — the Web activity payload limit could turn a truncation bug into a hard failure on the very dataset in the issue, and ARG paging without a sort can duplicate and drop rows.

[Roland Krummenacher (RolandKrummenacher)]

Heads-up: the five "Implemented" replies (Sep 21) aren't on the branch. xe/issue-2320-ef877f is still at bed630f from Sep 16 with no push since, so the commit may be sitting unpushed in a local worktree. Could you push it? I'll re-review then.

Threads 1 (output limit) and 3 (early loop exit) are the blockers, since both can silently drop data.

For the sort fix in thread 2: this module is generic, so a hardcoded order by x_RecommendationId would break future ARG queries without that column. Can the sort key come from the query config?

The PR also conflicts with dev (docs-mslearn/toolkit/changelog.md only); please merge origin/dev.

[Roland Krummenacher (RolandKrummenacher)]

Re-review — PR #2330 (head b9b7805)

All five review fixes are in b9b7805, and I checked each one in app.bicep:

• $top 500 is sent on every page, and $skipToken only when there is one.
• | order by x_RecommendationId asc comes before the source columns are added. Every shipped query in Recommendations/queries returns x_RecommendationId, and none uses take, limit, or sample, so Resource Graph keeps returning a $skipToken.
• hasMorePages depends only on the token, and hasResults only controls the write. An empty page that still has a token keeps the loop going without writing a file.
• The loop timeout is 6h, and Write ARG Page retries twice. A retry overwrites the same page file, so it doesn't create duplicates.
• replace(ingestionPath, '.parquet', …) replaces the substring arithmetic.

The module builds cleanly. Approving.

Documentation follow-ups (non-blocking)

1. Engine README: AzureResourceGraph/README.md still describes 1,000-row responses. Please mention the 500-row $top page size and the forced order by x_RecommendationId.
2. Sort column is fixed: The engine is a generic ExecuteQuery implementation, but it now requires x_RecommendationId in every query result. That's fine today, because custom queries must use the Recommendations dataset and x_RecommendationId is a required column. A query for any other dataset would fail, so the README should say this, or the sort column could become a parameter later.
3. Custom query tips: The pipeline doesn't check resultTruncated. If a custom query uses take, limit, or sample, Resource Graph returns no $skipToken, and only the first page is written without a warning. A one-line tip in Add custom recommendations would cover it.

Also note that the PR deployment jobs were skipped on this head, so CI hasn't run the paging loop end to end.