diff --git a/LICENSE b/LICENSE index 5edc644cf0eed..e034c879dac64 100644 --- a/LICENSE +++ b/LICENSE @@ -13,7 +13,7 @@ Business Source License 1.1 Licensor: Materialize, Inc. -Licensed Work: Materialize Version 20260903 +Licensed Work: Materialize Version 20260904 The Licensed Work is © 2026 Materialize, Inc. Additional Use Grant: Within a single installation of the Licensed Work, @@ -39,7 +39,7 @@ Additional Use Grant: Within a single installation of the Licensed Work, whose definitions are controlled by such third parties. -Change Date: September 03, 2030 +Change Date: September 04, 2030 Change License: Apache License, Version 2.0 diff --git a/console/AGENTS.md b/console/AGENTS.md index aa30845198a21..186e37235dd10 100644 --- a/console/AGENTS.md +++ b/console/AGENTS.md @@ -77,7 +77,12 @@ Use `~/` for imports from src: `import { foo } from "~/api/materialize"` ### State Management - **Jotai** for global state (sparingly used - prefer component state) -- **React Query** for server state +- **React Query** for polled/one-shot server state (`useSql` hooks) +- **Sync engine** for live catalog data: SUBSCRIBE-fed sessions reduce into + jotai atoms and TanStack DB collections (`useLiveQuery` consumers, scoped + instant-load cache). See `doc/design/20260902_sync_engine.md`. Wiring between + the engine's singletons belongs in session objects on the jotai store + (`store.sub`), not in React effects. - **Kysely** as query builder (NOT as database driver) for type-safe SQL against Materialize system catalog ### Stacks diff --git a/console/doc/architecture.md b/console/doc/architecture.md index 6c6d20aedb2e9..623b4356fdce8 100644 --- a/console/doc/architecture.md +++ b/console/doc/architecture.md @@ -69,3 +69,11 @@ state is stored in components with the useState hook. Storing state globally can be tricky to get right, in particular when it comes to invalidating / cleaning up the state. With component state, we get that behavior for free because the component gets unmounted. + +Server state is split by shape. Request/response data (polled telemetry, +one-shot queries, mutations) goes through React Query and the `useSql` hooks. +Live catalog data goes through the sync engine: app-session `SUBSCRIBE`s feed +subscribe sessions, which reduce into jotai atoms and TanStack DB collections, +with a per-tenant localStorage cache for instant load. See +[the sync engine design doc](design/20260902_sync_engine.md) for the full +architecture and adoption recipes. diff --git a/console/doc/design/20260902_sync_engine.md b/console/doc/design/20260902_sync_engine.md new file mode 100644 index 0000000000000..88213bba869b0 --- /dev/null +++ b/console/doc/design/20260902_sync_engine.md @@ -0,0 +1,290 @@ +# The sync engine + +- Associated: MaterializeInc/materialize#38231, MaterializeInc/materialize#38631 + +## Overview + +The console keeps a live, client-side replica of parts of the Materialize +catalog. The sync engine is the stack that maintains that replica: a +Materialize `SUBSCRIBE` feeds a reduction layer, plain session objects manage +lifecycle, and the results land in one of two stores that pages read from. The +newer of the two stores is TanStack DB, which adds incrementally maintained +live queries and a per-tenant instant-load cache on top of the same feed. + +Pages never see the plumbing. They read a jotai atom or run a `useLiveQuery` +against a collection, and the data is simply there, current, across +reconnects and region switches. + +``` + Materialize + SUBSCRIBE ... WITH (PROGRESS) ENVELOPE UPSERT (one per data set) + | + v WebSocket diff stream + SubscribeManager folds diffs into the current row set, + | emits one snapshot per closed timestamp + v + subscribe session plain object: lifecycle, region reset, + | cache hydration, sink writes + v + +----+---------------------------+ + | | + v v + jotai atom TanStack DB collection <---> localStorage cache + (SubscribeState) (rows diffed in) (scoped, instant load) + | | + v v + useAtomValue useLiveQuery + consumers consumers (incremental) +``` + +## Why two stores + +The jotai atoms (`allObjects`, `allSchemas`, `allClusters`, `allRoles`) hold +each data set as one array plus status. They are simple and fine for consumers +that render the whole set or run a cheap filter. + +TanStack DB collections hold the same rows keyed, with a live-query engine on +top. A collection is the right store when a surface needs any of: + +- **Incremental recomputation.** A live query (filters, joins, lookups) + recomputes from row deltas instead of re-scanning the full array on every + catalog tick. With atoms, every tick replaces the array and every consumer + re-filters and re-renders. +- **Fine-grained re-renders.** A component watching one key re-renders only + when that row changes, not whenever anything in the set changes. +- **Instant load.** Collections persist complete snapshots to a per-tenant + localStorage cache and seed from it on the next visit, so the surface + renders before the live snapshot arrives. + +Both stores are sinks of the same engine. Adopting a collection for a data set +does not change what the server sees: within a tab, the upstream SUBSCRIBE +count stays the same, and a collection can even be fed from an existing atom +(see the atom-fed session below) at zero additional upstream cost. + +## The layers + +| Layer | File | Owns | +| --- | --- | --- | +| `MaterializeWebsocket` | `api/materialize/MaterializeWebsocket.ts` | The wire: one WebSocket to the SQL HTTP endpoint, one subscribe per connection. | +| `SubscribeManager` | `api/materialize/SubscribeManager.ts` | The SUBSCRIBE protocol: buffers rows per timestamp, folds the upsert stream into the current row set, exposes snapshots via `onChange`/`getSnapshot`. Holds its last snapshot through a resubscribe so the UI does not flash empty. | +| `WebsocketConnectionManager` | `api/materialize/WebsocketConnectionManager.ts` | Connection lifecycle: subscribes to environment health and the current region on the jotai store, connects when healthy, reconnects with backoff, reconnects when the region's address changes, and tears down a socket left pointing at a departed region. | +| Subscribe sessions | `api/materialize/subscribeSession.ts` | Composition: one constructor wires the manager, the connection manager, the sink, the region reset, and cache hydration, in one defined order. | +| Hooks | `api/materialize/useSubscribe.ts` | Lifecycle adapters: a single effect creates the session on mount and destroys it on unmount. | +| Collection bridge | `api/materialize/subscribeCollection.ts` | `createSubscribeCollection`: turns manager snapshots into incremental insert/update/delete writes on a TanStack DB collection, owns the scoped localStorage cache, and mirrors status (error, snapshotComplete) into a companion atom. | +| Cache scope | `store/syncEngineCache.ts` | Derives the cache scope `organizationId\|regionId` as an atom, so hydration can be driven from the store. | + +The design rule behind the split: reactive plumbing between module-level +singletons belongs on the jotai store (`store.sub`), not in React effects. +Effects impose ordering between independently registered callbacks, re-run +under Suspense, and force every cross-cutting concern to be threaded into each +hook variant separately. A session wires everything once, synchronously, and +is testable without a DOM. + +## Subscribe sessions + +A session is a plain object created by a hook's single effect and destroyed on +unmount. Three kinds exist: + +- `createAtomSubscribeSession`: owns a socket, reduces into a jotai atom. + Holds existing atom data through a fresh connection's empty pre-snapshot, + and resets the manager and the atom to the loading state when the region + changes, so no page serves another region's catalog. +- `createCollectionSubscribeSession`: owns a socket, feeds a collection. + Additionally holds a keep-alive subscriber so the collection is not garbage + collected while no component queries it, clears the collection synchronously + on a region change, and hydrates it from its scoped cache via the scope atom. +- `createAtomFedCollectionSession`: no socket. Bridges an already running + subscribe atom into a collection, adding no upstream load, plus the same + region clear and scoped hydration. (The source atom's own reset reaches the + bridge as an empty pre-snapshot, which applySnapshot ignores, so the + collection must clear itself.) + +``` + +--------------------------------------+ + hook | subscribe session (plain object) | + one effect: | | +-----------+ + create / destroy --+-> SubscribeManager <---- diff stream +----- | WebSocket | + | WebsocketConnectionManager --------+----> +-----------+ + | reconnect at addr| + | sink: atom | collection -----------+----> atoms / collections + | | | + | store.sub(regionId) -> reset | v + | store.sub(cacheScope) -> hydrate | localStorage cache + +--------------------------------------+ (scoped persist) +``` + +## The collection bridge + +`createSubscribeCollection` is the seam between the feed and TanStack DB. The +manager already reduces the diff stream to the current row set, so the bridge +diffs each snapshot against what the collection holds and writes only the +delta as insert/update/delete operations. Live queries downstream therefore +recompute incrementally. + +The bridge also owns the instant-load cache: + +- **Persist.** With a `persistName`, complete snapshots (never partial state) + are written to localStorage on a trailing throttle, under the key + `mz-console:sync-engine:|||v`. +- **Hydrate.** Seeding is deferred until the scope is known, then the cached + rows load into the collection and the loading gate opens immediately. A + cached snapshot counts as complete for gating; the live snapshot replaces it + when it arrives. Other scopes' cache entries are pruned on hydrate. +- **Fail closed.** A scope resolution that errors suspends persistence + entirely rather than leaving writes aimed at the previous scope's key. +- **Scope changes.** Re-hydrating under a new scope drops the in-memory rows + and any pending persist first, so one tenant's rows are never shown as, or + written under, another tenant's cache. +- **Status.** Error and snapshotComplete mirror into a companion jotai atom so + consumers keep the same loading/error semantics as atom-backed data. + +An empty pre-snapshot from a fresh connection carries no data: the bridge +ignores it rather than clearing cache-seeded state or counting it as live +data, except that it clears a stale error so the UI falls back to loading +during reconnect backoff. + +## Data flow + +Steady state: the socket delivers the diff stream, the manager folds it and +emits a snapshot per closed timestamp, the sink writes it into the atom or +diffs it into the collection, and `useLiveQuery` consumers recompute +incrementally. + +On a region switch: + +1. The region atom is written by the region selector. +2. The connection manager sees the new address and reconnects the socket + there. Pausing on an unhealthy target tears the departed region's socket + down, so a later return to a healthy region always resumes. A health blip + within the current region leaves a working socket alone. +3. Each session's region subscription fires synchronously: the manager drops + its held rows and atoms go back to loading. +4. The cache scope atom recomputes. Collection sessions hydrate the new + scope's cache, dropping in-memory rows and any pending persist. +5. The new region's snapshot arrives and flows through the sinks as usual. + +On a reconnect within one region, the manager keeps its last snapshot and +re-emits it when the socket reopens, so the UI holds data through the +resubscribe instead of flashing empty. + +## Boundaries + +What the engine is not: + +- **Not a server-load lever by itself.** It is a client store. Within one tab + the upstream SUBSCRIBE count is unchanged by adopting it. Its sessions and + sinks are transport-agnostic, which is the seam a shared upstream + (cross-tab or cross-client dedup) can plug into without touching surfaces, + but the engine alone does not provide that. +- **Not for polled telemetry.** Expensive introspection queries + (utilization history, arrangement sizes) are react-query territory, with + polling intervals and its persistence tooling. The engine serves + catalog-shaped data that a SUBSCRIBE can maintain. +- **Not for page-scoped, request-driven subscribes.** Those use + `useSubscribe`/`useSubscribeManager`, which tie a manager to a component + and expose state via `useSyncExternalStore`. + +## Adopting a collection for a surface + +A surface that moves from an atom read to a collection gets, without further +code: incremental live queries, per-key re-render granularity, instant load +from the scoped cache, keep-alive for the app session, and region-correct +resets. The object explorer is the reference adoption: its tree runs live +queries over the objects collection (fed from the app-wide `allObjects` atom) +and the namespaces collection (its own subscribe), and renders from cache on +revisit before the live snapshot lands. + +Declare configuration once at module scope. The hooks require a referentially +stable options object, and a new object identity restarts the session, which +makes accidental instability visible instead of masked. + +Atom-backed data set: + +```ts +export const allWidgets = atom>({ + data: [], + error: undefined, + snapshotComplete: false, +}); + +const ALL_WIDGETS_SUBSCRIBE_OPTIONS = { + atom: allWidgets, + subscribe: buildSubscribeQuery(buildWidgetsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; + +export function useSubscribeToAllWidgets() { + useGlobalUpsertSubscribe(ALL_WIDGETS_SUBSCRIBE_OPTIONS); +} +``` + +Collection with its own subscribe and the instant-load cache: + +```ts +export const widgetsCollection = createSubscribeCollection({ + id: "widgets", + getKey: (widget) => widget.id, + persistName: "widgets", +}); + +const WIDGETS_SUBSCRIBE_OPTIONS = { + target: widgetsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + subscribe: buildSubscribeQuery(buildWidgetsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; + +export function useSubscribeToWidgetsCollection() { + useGlobalSubscribeCollection(WIDGETS_SUBSCRIBE_OPTIONS); +} +``` + +Collection fed from an existing atom, no second socket: + +```ts +export const widgetsCollection = createSubscribeCollection({ + id: "widgets", + getKey: (widget) => widget.id, + persistName: "widgets", +}); + +export function useSubscribeToWidgetsCollection() { + const store = useStore(); + React.useEffect(() => { + const session = createAtomFedCollectionSession({ + store, + sourceAtom: allWidgets, + target: widgetsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + }); + return session.destroy; + }, [store]); +} +``` + +Consumers read collections through `useLiveQuery`, with status from the +companion atom: + +```ts +const { data } = useLiveQuery((q) => q.from({ widgets: widgetsCollection.collection })); +const status = useAtomValue(widgetsCollection.statusAtom); +``` + +Mount the activator hook once: in `AppInitializer` for data the whole app +reads, or in a route component for data one surface reads. Note that a route +component keyed by a path pattern survives param-only navigation, including +region switches, which is why region handling lives in the session rather +than in mount/unmount. + +## Testing + +Sessions and the collection bridge are plain objects, so their behavior is +asserted headlessly and synchronously: create them against `getStore()`, +write the region or scope atoms or call `applySnapshot`, and assert on the +sink in the same tick. No rendering, no Suspense, no `waitFor`. See +`useSubscribe.test.tsx` and `subscribeCollection.test.ts`. jsdom-rendered +tests of this stack are unreliable (sockets fail and produce error states, +Suspense tears sessions down mid-switch), so prefer session-level tests plus +component tests that mock the activator hooks. diff --git a/console/package.json b/console/package.json index 0974b9a412806..d5992e98df7be 100644 --- a/console/package.json +++ b/console/package.json @@ -239,6 +239,17 @@ "prettier/prettier": "error", "simple-import-sort/imports": "error", "simple-import-sort/exports": "error", + "no-restricted-syntax": [ + "error", + { + "selector": "CallExpression[callee.name='useGlobalUpsertSubscribe'] > ObjectExpression", + "message": "Declare subscribe options at module scope; an inline object restarts the SUBSCRIBE session on every render." + }, + { + "selector": "CallExpression[callee.name='useGlobalSubscribeCollection'] > ObjectExpression", + "message": "Declare subscribe options at module scope; an inline object restarts the SUBSCRIBE session on every render." + } + ], "@typescript-eslint/no-shadow": "error", "@typescript-eslint/ban-ts-comment": "off", "@typescript-eslint/explicit-module-boundary-types": "off", diff --git a/console/src/api/auth.tsx b/console/src/api/auth.tsx index de32b3e02d1ea..c94ca8578b99d 100644 --- a/console/src/api/auth.tsx +++ b/console/src/api/auth.tsx @@ -14,6 +14,7 @@ import { useQuery } from "@tanstack/react-query"; +import { AppConfig } from "~/config/AppConfig"; import { CloudRuntimeConfig } from "~/config/AppConfigSwitch"; import { useAppConfig } from "~/config/useAppConfig"; import { @@ -170,6 +171,20 @@ export function useSelfManagedSubscription() { return { subscription, isLoading, isError, error }; } +/** + * Whether this deployment fetches the current organization at all. This also + * decides the organization half of the sync-engine cache key + * (store/syncEngineCache.ts), which isolates tenants' caches on a shared + * origin, so every caller must use this one predicate. + */ +export function isOrganizationFetchEnabled(appConfig: AppConfig): boolean { + const isLocalImpersonation = + appConfig.mode === "cloud" && + appConfig.isImpersonating && + appConfig.isLocalImpersonation; + return appConfig.mode !== "self-managed" && !isLocalImpersonation; +} + // Convenience hook for getting the current organization ID based on // the app's deployment mode. Although "organizations" don't really exist in // self managed environments, a lot of our code uses the organization ID as part of a key to cache @@ -177,13 +192,8 @@ export function useSelfManagedSubscription() { // in deployment modes where the organization ID is not available. export function useMaybeCurrentOrganizationId() { const appConfig = useAppConfig(); - const isLocalImpersonation = - appConfig.mode === "cloud" && - appConfig.isImpersonating && - appConfig.isLocalImpersonation; - const isCurrentOrganizationFetchEnabled = - appConfig.mode !== "self-managed" && !isLocalImpersonation; + isOrganizationFetchEnabled(appConfig); const res = useQuery({ queryKey: queryKeys.currentOrganization(), diff --git a/console/src/api/materialize/WebsocketConnectionManager.test.ts b/console/src/api/materialize/WebsocketConnectionManager.test.ts index f71bd63f37659..4fb59194b5775 100644 --- a/console/src/api/materialize/WebsocketConnectionManager.test.ts +++ b/console/src/api/materialize/WebsocketConnectionManager.test.ts @@ -12,9 +12,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { getStore } from "~/jotai"; import { + currentRegionIdAtom, EnvironmentsWithHealth, environmentsWithHealth, } from "~/store/environments"; +import { defaultRegionId } from "~/test/utils"; import { Connectable, @@ -234,6 +236,169 @@ describe("WebsocketConnectionManager", () => { }); }); + describe("region switch", () => { + const twoRegions = () => + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", createHealthyEnvironment("addr-b:6876")], + ]) as unknown as EnvironmentsWithHealth; + + afterEach(() => { + getStore().set(currentRegionIdAtom, defaultRegionId); + }); + + it("reconnects at the new address when the region switches while connected", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + store.set(currentRegionIdAtom, "aws/eu-west-1"); + + expect(mockTarget.reconnect).toHaveBeenCalledTimes(1); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-b:6876", + undefined, + ); + }); + + it("reconnects after a round trip through an unhealthy region", () => { + const store = getStore(); + const crashed = { + ...createHealthyEnvironment("addr-b:6876"), + status: { health: "crashed" as const, version: "0.0.0", errors: [] }, + }; + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", crashed], + ]) as unknown as EnvironmentsWithHealth, + ); + + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + // Pausing on the unhealthy region tears the old region's socket down. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.disconnect).toHaveBeenCalled(); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + + // Returning to the healthy region reconnects rather than assuming the + // old socket still serves it. + store.set(currentRegionIdAtom, defaultRegionId); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + }); + + it("keeps the socket through a same-region health blip", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + vi.mocked(mockTarget.disconnect).mockClear(); + + // The current region's health check fails: the socket is left alone, + // so consumers like the Shell see no silent teardown. + store.set( + environmentsWithHealth, + new Map([ + [ + defaultRegionId, + { + ...createHealthyEnvironment("addr-a:6876"), + status: { + health: "crashed" as const, + version: "0.0.0", + errors: [], + }, + }, + ], + ["aws/eu-west-1", createHealthyEnvironment("addr-b:6876")], + ]) as unknown as EnvironmentsWithHealth, + ); + expect(mockTarget.disconnect).not.toHaveBeenCalled(); + + // Health recovers: the socket is still connected, so no churn either. + store.set(environmentsWithHealth, twoRegions()); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + }); + + it("reconnects after a round trip through a disabled region", () => { + const store = getStore(); + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, createHealthyEnvironment("addr-a:6876")], + ["aws/eu-west-1", { state: "disabled" as const }], + ]) as unknown as EnvironmentsWithHealth, + ); + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + mockTarget.simulateOpen(); + vi.mocked(mockTarget.reconnect).mockClear(); + + // A disabled region never updates the address, so the teardown must key + // on the region, not the address. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.disconnect).toHaveBeenCalled(); + + store.set(currentRegionIdAtom, defaultRegionId); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + }); + + it("reconnects after a region switch that lands mid-handshake", () => { + const store = getStore(); + store.set(environmentsWithHealth, twoRegions()); + + // Constructor starts a connect to region A; handshake still in flight. + manager = new WebsocketConnectionManager( + mockTarget, + store, + reconnectionStateAtom, + ); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-a:6876", + undefined, + ); + vi.mocked(mockTarget.reconnect).mockClear(); + + // Switch regions before the handshake completes: deferred, not dropped. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(mockTarget.reconnect).not.toHaveBeenCalled(); + + mockTarget.simulateOpen(); + expect(mockTarget.reconnect).toHaveBeenCalledWith( + "addr-b:6876", + undefined, + ); + }); + }); + describe("destroy", () => { it("disconnects and cleans up subscriptions", () => { const store = getStore(); diff --git a/console/src/api/materialize/WebsocketConnectionManager.ts b/console/src/api/materialize/WebsocketConnectionManager.ts index e6884a62a46cd..f766224246d76 100644 --- a/console/src/api/materialize/WebsocketConnectionManager.ts +++ b/console/src/api/materialize/WebsocketConnectionManager.ts @@ -77,6 +77,10 @@ export class WebsocketConnectionManager { private isHealthy = false; private currentHttpAddress?: string; + /** Address used for the most recent connection attempt. */ + private attemptedHttpAddress?: string; + /** Region the most recent connection attempt was made for. */ + private attemptedRegionId?: string | null; private retryAttempt = 0; private hasEverConnected = false; private retryTimer: ReturnType | undefined; @@ -158,11 +162,20 @@ export class WebsocketConnectionManager { } if (nowHealthy) { - if (!this.target.isConnected()) { + // A connected socket may still stream from a previous region's address + // after a region switch; reconnect at the current one. + if ( + !this.target.isConnected() || + this.attemptedHttpAddress !== this.currentHttpAddress + ) { this.resumeConnection(); } } else { - this.pauseConnection(); + // Only tear down a socket pointing at a region the user has left; a + // health blip in the current region leaves a working socket alone + // (consumers like the Shell would get no close callback). + const regionId = this.store.get(currentRegionIdSyncAtom); + this.pauseConnection(regionId !== this.attemptedRegionId); } }; @@ -205,6 +218,14 @@ export class WebsocketConnectionManager { this.retryAttempt = 0; this.clearRetryTimer(); this.notifyStateChange(); + // The region may have switched while this handshake was in flight; the + // socket that just opened points at the old address. + if ( + this.isHealthy && + this.attemptedHttpAddress !== this.currentHttpAddress + ) { + this.attemptConnection(); + } }; /** Tear down and reopen the socket. Used on SQL request changes. */ @@ -239,6 +260,8 @@ export class WebsocketConnectionManager { hasEverConnected: this.hasEverConnected, }); this.connectInFlight = true; + this.attemptedHttpAddress = this.currentHttpAddress; + this.attemptedRegionId = this.store.get(currentRegionIdSyncAtom); try { this.target.reconnect(this.currentHttpAddress, sessionVariables); } catch { @@ -253,9 +276,15 @@ export class WebsocketConnectionManager { this.attemptConnection(); } - private pauseConnection() { + private pauseConnection(disconnect: boolean) { this.clearRetryTimer(); this.retryAttempt = 0; + if (disconnect) { + // Tear down so the resume path reconnects after an unhealthy round trip; + // clear connectInFlight since this socket gets no open/close callback. + this.connectInFlight = false; + this.target.disconnect(); + } this.notifyStateChange(); } diff --git a/console/src/api/materialize/subscribeCollection.test.ts b/console/src/api/materialize/subscribeCollection.test.ts index 8020b3ec04252..0372c7894146d 100644 --- a/console/src/api/materialize/subscribeCollection.test.ts +++ b/console/src/api/materialize/subscribeCollection.test.ts @@ -105,9 +105,8 @@ describe("createSubscribeCollection", () => { }); it("hydrates from cache after an empty pre-snapshot was applied", async () => { - // The mount effect pushes the atom's empty initial state before the async - // auth/region scope resolves, so hydrate always runs after it. The empty - // placeholder must not count as live data, or the cache is never read. + // The empty initial state is pushed before the async scope resolves, so + // hydrate always runs after it; it must not count as live data. const name = "late-hydrate"; const scope = "org1|region1"; localStorage.setItem( @@ -143,6 +142,121 @@ describe("createSubscribeCollection", () => { expect(collection.size).toBe(1); }); + it("does not carry the previous scope's rows into a new scope", async () => { + const name = "scope-switch"; + const keyB = syncEngineCacheKey(name, "org|regionB"); + const { collection, statusAtom, applySnapshot, hydrate } = + freshCollection(name); + + hydrate("org|regionA"); + applySnapshot(state([{ id: "1", name: "region-a" }])); + await flush(); + expect(collection.size).toBe(1); + + vi.useFakeTimers(); + try { + // A region switch re-hydrates with a new scope while region A's rows are + // still in memory. They must not be persisted under region B's key. + hydrate("org|regionB"); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(keyB)).toBeNull(); + } finally { + vi.useRealTimers(); + } + + // Nor served as region B data: the collection empties and the loading + // gate closes until region B's own snapshot arrives. + await flush(); + expect(collection.size).toBe(0); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + }); + + it("seeds the new scope from its own cache after a scope switch", async () => { + const name = "scope-switch-seed"; + const keyB = syncEngineCacheKey(name, "org|regionB"); + const { collection, applySnapshot, hydrate } = freshCollection(name); + + hydrate("org|regionA"); + applySnapshot(state([{ id: "1", name: "region-a" }])); + await flush(); + + // Written after region A's hydrate pruned other scopes, as another tab + // already on region B would. + localStorage.setItem(keyB, JSON.stringify([{ id: "9", name: "region-b" }])); + hydrate("org|regionB"); + await flush(); + + expect(collection.has("1")).toBe(false); + expect(collection.get("9")?.name).toBe("region-b"); + }); + + it("reset drops rows, pending persistence, and the loading gate", async () => { + vi.useFakeTimers(); + try { + const name = "reset"; + const scope = "org|regionA"; + const key = syncEngineCacheKey(name, scope); + const { collection, statusAtom, applySnapshot, hydrate, reset } = + freshCollection(name); + hydrate(scope); + applySnapshot(state([{ id: "1", name: "a" }])); + + // Reset before the persist throttle fires: nothing may be written. + reset(); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + + vi.useRealTimers(); + await flush(); + expect(collection.size).toBe(0); + } finally { + vi.useRealTimers(); + } + }); + + it("suspendPersistence stops writes until a scope resolves again", () => { + vi.useFakeTimers(); + try { + const name = "suspend"; + const scope = "org|regionA"; + const key = syncEngineCacheKey(name, scope); + const { applySnapshot, hydrate, suspendPersistence } = + freshCollection(name); + hydrate(scope); + applySnapshot(state([{ id: "1", name: "a" }])); + + // Suspend cancels the pending write and later snapshots stay unpersisted. + suspendPersistence(); + applySnapshot(state([{ id: "2", name: "b" }])); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + + // A scope resolving again re-arms persistence, even the same scope. + hydrate(scope); + applySnapshot(state([{ id: "3", name: "c" }])); + vi.advanceTimersByTime(1100); + expect(JSON.parse(localStorage.getItem(key) ?? "[]")).toEqual([ + { id: "3", name: "c" }, + ]); + } finally { + vi.useRealTimers(); + } + }); + + it("clears a stale error when an empty pre-snapshot follows a reconnect", () => { + const { applySnapshot, statusAtom } = freshCollection(); + const error = { code: "boom", message: "it failed" }; + applySnapshot(state([], false, error)); + expect(getStore().get(statusAtom).error).toEqual(error); + + // SubscribeManager re-emits an empty pre-snapshot when a reconnect opens; + // the status must fall back to loading, not hold the error. + applySnapshot(state([], false)); + expect(getStore().get(statusAtom).error).toBeUndefined(); + expect(getStore().get(statusAtom).snapshotComplete).toBe(false); + }); + it("surfaces error and snapshotComplete on the status atom", () => { const { applySnapshot, statusAtom } = freshCollection(); diff --git a/console/src/api/materialize/subscribeCollection.ts b/console/src/api/materialize/subscribeCollection.ts index f234f292e2303..c8c79691cfe24 100644 --- a/console/src/api/materialize/subscribeCollection.ts +++ b/console/src/api/materialize/subscribeCollection.ts @@ -72,6 +72,14 @@ export interface SubscribeCollection { * tenant's catalog into another's session after an org or region switch. Only * the first call per scope seeds; later calls for the same scope no-op. */ hydrate: (scope: string) => void; + /** Drop all rows and pending persistence and return to the loading state, + * for when the held rows belong to another region's catalog. A later + * hydrate may still seed the new scope from its cache. */ + reset: () => void; + /** Stop persisting until a scope resolves again. Scope resolution failing + * must fail closed: writes aimed at a previous scope's key would store one + * tenant's rows under another's cache. */ + suspendPersistence: () => void; } /** @@ -213,14 +221,15 @@ export function createSubscribeCollection(options: { }; const applySnapshot = (state: SubscribeState) => { - // An empty pre-snapshot carries no information: ignore it entirely, so it - // neither clears cache-seeded state nor counts as live data. Marking it - // live would block a later hydrate (the scope resolves asynchronously, so - // hydrate always runs after the first empty push) and leave the cache - // permanently unread. + // An empty pre-snapshot carries no data: don't clear cache-seeded state or + // count it as live (that would block the later, async-scoped hydrate). It + // does clear a stale error so a reconnect falls back to loading. const isEmptyPreload = !state.snapshotComplete && state.data.length === 0 && !state.error; - if (isEmptyPreload) return; + if (isEmptyPreload) { + writeStatus({ error: undefined, snapshotComplete: lastSnapshotComplete }); + return; + } liveSnapshotApplied = true; desired = new Map(state.data.map((row) => [getKey(row), row])); @@ -254,9 +263,21 @@ export function createSubscribeCollection(options: { const hydrate = (scope: string) => { if (!persistName || hydratedScope === scope) return; + const scopeChanged = hydratedScope !== undefined; hydratedScope = scope; persistKey = syncEngineCacheKey(persistName, scope); pruneOtherScopes(persistKey); + if (scopeChanged) { + // The rows in memory belong to the previous scope: drop them and the + // pending persist (its timer reads `desired` at fire time). + if (persistTimer) { + clearTimeout(persistTimer); + persistTimer = undefined; + } + desired = new Map(); + liveSnapshotApplied = false; + lastSnapshotComplete = false; + } if (liveSnapshotApplied) { // Live data already arrived; don't overwrite it with the cache, but do // persist it now that the scoped key is known. @@ -264,14 +285,50 @@ export function createSubscribeCollection(options: { return; } const cached = readPersistedRows(persistKey); - if (!cached.length) return; - desired = new Map(cached.map((row) => [getKey(row), row])); - // A cached full snapshot counts as complete for the loading gate; the live - // snapshot refreshes it once it arrives. - lastSnapshotComplete = true; + if (cached.length) { + desired = new Map(cached.map((row) => [getKey(row), row])); + // A cached full snapshot counts as complete for the loading gate; the + // live snapshot refreshes it once it arrives. + lastSnapshotComplete = true; + } else if (!scopeChanged) { + return; + } + // On a scope change this also flushes the emptied set, so the collection + // stops serving the previous scope's rows while the new snapshot loads. + materialize(); + writeStatus({ error: undefined, snapshotComplete: lastSnapshotComplete }); + }; + + const clearPendingPersist = () => { + if (persistTimer) { + clearTimeout(persistTimer); + persistTimer = undefined; + } + }; + + const reset = () => { + clearPendingPersist(); + desired = new Map(); + lastSnapshotComplete = false; + liveSnapshotApplied = false; materialize(); - writeStatus({ error: undefined, snapshotComplete: true }); + writeStatus({ error: undefined, snapshotComplete: false }); + }; + + const suspendPersistence = () => { + clearPendingPersist(); + persistKey = undefined; + // Allow a later successful resolution to hydrate again, even for the + // scope whose resolution just failed. + hydratedScope = undefined; }; - return { collection, statusAtom, applySnapshot, hydrate }; + return { + collection, + statusAtom, + applySnapshot, + hydrate, + reset, + suspendPersistence, + }; } diff --git a/console/src/api/materialize/subscribeSession.ts b/console/src/api/materialize/subscribeSession.ts new file mode 100644 index 0000000000000..6481746a2ebc6 --- /dev/null +++ b/console/src/api/materialize/subscribeSession.ts @@ -0,0 +1,222 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Atom, createStore, PrimitiveAtom } from "jotai"; + +import { reconnectionStateAtom } from "~/hooks/useAutomaticallyConnectSocket"; +import { currentRegionIdSyncAtom } from "~/store/environments"; + +import { SubscribeCollection } from "./subscribeCollection"; +import { + SelectFunction, + SubscribeManager, + SubscribeRow, + SubscribeState, + UpsertKeyFunction, +} from "./SubscribeManager"; +import { SessionVariables, SqlRequest } from "./types"; +import { WebsocketConnectionManager } from "./WebsocketConnectionManager"; + +type JotaiStore = ReturnType; + +/** The shape jotai's `loadable()` wraps an async atom into. */ +export type ScopeLoadableAtom = Atom< + | { state: "loading" } + | { state: "hasError"; error: unknown } + | { state: "hasData"; data: string | undefined } +>; + +/** Hydrates `target` from its scoped cache whenever the scope settles or + * changes; hydrate itself handles both seeding and the scope-change reset. A + * failed scope resolution fails closed: persistence is suspended rather than + * left aimed at a previous scope's key. */ +function subscribeToScope( + store: JotaiStore, + scopeAtom: ScopeLoadableAtom, + target: { + hydrate: (scope: string) => void; + suspendPersistence: () => void; + }, +) { + const applyScope = () => { + const value = store.get(scopeAtom); + if (value.state === "hasData" && value.data) target.hydrate(value.data); + else if (value.state === "hasError") target.suspendPersistence(); + }; + applyScope(); + return store.sub(scopeAtom, applyScope); +} + +/** Invokes `onRegionChange` when the current region changes. */ +function subscribeToRegionChange( + store: JotaiStore, + onRegionChange: () => void, +) { + let regionId = store.get(currentRegionIdSyncAtom); + return store.sub(currentRegionIdSyncAtom, () => { + const next = store.get(currentRegionIdSyncAtom); + if (next === regionId) return; + regionId = next; + onRegionChange(); + }); +} + +export interface SubscribeSessionOptions { + store: JotaiStore; + request: SqlRequest; + upsertKey: UpsertKeyFunction; + select?: SelectFunction; + sessionVariables?: SessionVariables; + closeSocketOnComplete?: boolean; +} + +export interface SubscribeSession { + manager: SubscribeManager; + destroy: () => void; +} + +/** + * One app-session subscribe, wired outside React: a SubscribeManager kept + * connected by a WebsocketConnectionManager, snapshots pushed into `sink`, and + * a region-change reset subscribed directly on the store. Hooks only own the + * session's lifecycle; the composition is testable headlessly. + */ +function createSubscribeSession( + options: SubscribeSessionOptions, + sink: { + apply: (snapshot: SubscribeState) => void; + onRegionChange?: () => void; + }, +): SubscribeSession { + const { store, request, upsertKey, select } = options; + const manager = new SubscribeManager({ + request, + // The connection manager supplies the address on every attempt. + httpAddress: "", + upsert: { key: upsertKey }, + select, + sessionVariables: options.sessionVariables, + closeSocketOnComplete: options.closeSocketOnComplete, + }); + const unsubscribeChange = manager.onChange(() => + sink.apply(manager.getSnapshot()), + ); + const connection = new WebsocketConnectionManager( + manager, + store, + reconnectionStateAtom, + ); + + // The held rows belong to the previous region's catalog: reset rather than + // serve or replay them while the socket re-subscribes at the new address. + const unsubscribeRegion = subscribeToRegionChange(store, () => { + manager.reset(); + sink.onRegionChange?.(); + }); + + return { + manager, + destroy() { + unsubscribeRegion(); + unsubscribeChange(); + connection.destroy(); + }, + }; +} + +/** A subscribe session that reduces into a jotai atom (allObjects et al.). */ +export function createAtomSubscribeSession< + T extends object, + R = SubscribeRow, +>( + options: SubscribeSessionOptions & { + atom: PrimitiveAtom>; + }, +): SubscribeSession { + const { store, atom } = options; + return createSubscribeSession(options, { + apply: (snapshot) => { + const current = store.get(atom); + if (current === snapshot) return; + // Hold cached atom data through a fresh manager's empty pre-snapshot state. + const snapshotIsEmptyPreload = + !snapshot.snapshotComplete && !snapshot.data.length && !snapshot.error; + if (snapshotIsEmptyPreload && current.data.length) return; + store.set(atom, snapshot); + }, + onRegionChange: () => + store.set(atom, { data: [], snapshotComplete: false, error: undefined }), + }); +} + +/** A subscribe session that feeds a TanStack DB collection (namespaces et al.). + * The region change clears the collection synchronously; hydrate is + * responsible only for cache seeding. */ +export function createCollectionSubscribeSession< + T extends object, + R extends object = SubscribeRow, +>( + options: SubscribeSessionOptions & { + target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; + }, +): SubscribeSession { + const { store, target, scopeAtom } = options; + // Hold a subscriber for the session so the collection doesn't pause/GC + // while no component is querying it. + const keepAlive = target.collection.subscribeChanges(() => {}); + const unsubscribeScope = scopeAtom + ? subscribeToScope(store, scopeAtom, target) + : undefined; + const session = createSubscribeSession(options, { + apply: (snapshot) => target.applySnapshot(snapshot), + onRegionChange: () => target.reset(), + }); + target.applySnapshot(session.manager.getSnapshot()); + return { + manager: session.manager, + destroy() { + unsubscribeScope?.(); + keepAlive.unsubscribe(); + session.destroy(); + }, + }; +} + +/** + * Feeds a collection from an already-running subscribe atom rather than its + * own socket, adding no upstream load. The source atom's own session resets + * the atom on a region change, but that reaches this bridge as an empty + * pre-snapshot, which applySnapshot ignores, so the collection clears itself + * on the region change too. + */ +export function createAtomFedCollectionSession(options: { + store: JotaiStore; + sourceAtom: Atom>; + target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; +}): { destroy: () => void } { + const { store, sourceAtom, target, scopeAtom } = options; + const unsubscribeScope = scopeAtom + ? subscribeToScope(store, scopeAtom, target) + : undefined; + const unsubscribeRegion = subscribeToRegionChange(store, () => + target.reset(), + ); + const apply = () => target.applySnapshot(store.get(sourceAtom)); + apply(); + const unsubscribeSource = store.sub(sourceAtom, apply); + return { + destroy() { + unsubscribeScope?.(); + unsubscribeRegion(); + unsubscribeSource(); + }, + }; +} diff --git a/console/src/api/materialize/useSubscribe.test.tsx b/console/src/api/materialize/useSubscribe.test.tsx new file mode 100644 index 0000000000000..cbd432a011e02 --- /dev/null +++ b/console/src/api/materialize/useSubscribe.test.tsx @@ -0,0 +1,237 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { atom } from "jotai"; +import { ws } from "msw"; + +import server from "~/api/mocks/server"; +import { getStore } from "~/jotai"; +import { + currentEnvironmentState, + currentRegionIdAtom, + EnvironmentsWithHealth, + environmentsWithHealth, +} from "~/store/environments"; +import { defaultRegionId, healthyEnvironment } from "~/test/utils"; + +import { + createSubscribeCollection, + syncEngineCacheKey, +} from "./subscribeCollection"; +import { SubscribeState } from "./SubscribeManager"; +import { + createAtomFedCollectionSession, + createAtomSubscribeSession, + createCollectionSubscribeSession, +} from "./subscribeSession"; + +// Accept subscribe sockets and stay silent, so connections neither error nor +// deliver data: the only thing that may then change state is the session. +const subscribeSocket = ws.link("ws://*"); + +type Row = { id: string }; + +const request = { queries: [{ query: "SELECT 1", params: [] }] }; +const upsertKey = (row: { data: Row }) => row.data.id; + +/** Let the collection's async sync writes settle before asserting. */ +const flush = () => new Promise((resolve) => setTimeout(resolve, 0)); + +const loadingState: SubscribeState = { + data: [], + snapshotComplete: false, + error: undefined, +}; + +function setTwoRegions() { + const store = getStore(); + store.set( + environmentsWithHealth, + new Map([ + [defaultRegionId, { ...healthyEnvironment, httpAddress: "addr-a:6876" }], + ["aws/eu-west-1", { ...healthyEnvironment, httpAddress: "addr-b:6876" }], + ]) as unknown as EnvironmentsWithHealth, + ); + return store; +} + +describe("subscribeSession", () => { + beforeEach(() => { + server.use(subscribeSocket.addEventListener("connection", () => {})); + }); + + afterEach(async () => { + const store = getStore(); + store.set(currentRegionIdAtom, defaultRegionId); + await store.get(currentEnvironmentState); + }); + + it("resets the atom to loading only when the region actually changes", () => { + const store = setTwoRegions(); + const testAtom = atom>(loadingState); + const session = createAtomSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + atom: testAtom, + }); + try { + // A completed snapshot from the previous region is in the atom. + store.set(testAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + + // Re-setting the same region is not a change. + store.set(currentRegionIdAtom, defaultRegionId); + expect(store.get(testAtom).data).toEqual([{ id: "1" }]); + + // The previous region's rows must not be served as the new region's data. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(store.get(testAtom).data).toEqual([]); + expect(store.get(testAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("stops reacting to region changes after destroy", () => { + const store = setTwoRegions(); + const testAtom = atom>(loadingState); + const session = createAtomSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + atom: testAtom, + }); + session.destroy(); + + store.set(testAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + store.set(currentRegionIdAtom, "aws/eu-west-1"); + expect(store.get(testAtom).data).toEqual([{ id: "1" }]); + }); + + it("clears the collection when the region switches", async () => { + const store = setTwoRegions(); + const target = createSubscribeCollection({ + id: "session-test-collection", + getKey: (row) => row.id, + }); + const session = createCollectionSubscribeSession({ + store, + request, + upsertKey, + select: (row) => row.data, + target, + }); + try { + // A completed snapshot from the previous region is in the collection. + target.applySnapshot({ + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + await flush(); + expect(target.collection.size).toBe(1); + + // The rows must be gone and the loading gate closed, not merely a + // manager reset invoked: that reset reaches applySnapshot as an empty + // pre-snapshot, which it ignores by design. + store.set(currentRegionIdAtom, "aws/eu-west-1"); + await flush(); + expect(target.collection.size).toBe(0); + expect(store.get(target.statusAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("clears an atom-fed collection when the region switches", async () => { + const store = setTwoRegions(); + const sourceAtom = atom>(loadingState); + const target = createSubscribeCollection({ + id: "session-test-atom-fed", + getKey: (row) => row.id, + }); + const session = createAtomFedCollectionSession({ + store, + sourceAtom, + target, + }); + try { + store.set(sourceAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + await flush(); + expect(target.collection.size).toBe(1); + + store.set(currentRegionIdAtom, "aws/eu-west-1"); + await flush(); + expect(target.collection.size).toBe(0); + expect(store.get(target.statusAtom).snapshotComplete).toBe(false); + } finally { + session.destroy(); + } + }); + + it("suspends persistence when the scope fails to resolve", () => { + vi.useFakeTimers(); + const store = setTwoRegions(); + const scopeAtom = atom< + | { state: "loading" } + | { state: "hasError"; error: unknown } + | { state: "hasData"; data: string | undefined } + >({ state: "loading" }); + const sourceAtom = atom>(loadingState); + const target = createSubscribeCollection({ + id: "session-test-scope-error", + getKey: (row) => row.id, + persistName: "session-scope-error", + }); + const session = createAtomFedCollectionSession({ + store, + sourceAtom, + target, + scopeAtom, + }); + const key = syncEngineCacheKey("session-scope-error", "org|regionA"); + try { + store.set(scopeAtom, { state: "hasData", data: "org|regionA" }); + store.set(sourceAtom, { + data: [{ id: "1" }], + snapshotComplete: true, + error: undefined, + }); + + // Scope resolution fails before the persist throttle fires: nothing may + // be written under the last known scope's key. + store.set(scopeAtom, { state: "hasError", error: new Error("boom") }); + store.set(sourceAtom, { + data: [{ id: "2" }], + snapshotComplete: true, + error: undefined, + }); + vi.advanceTimersByTime(1100); + expect(localStorage.getItem(key)).toBeNull(); + } finally { + vi.useRealTimers(); + session.destroy(); + localStorage.clear(); + } + }); +}); diff --git a/console/src/api/materialize/useSubscribe.ts b/console/src/api/materialize/useSubscribe.ts index c79212b2fd199..50f87b319d92f 100644 --- a/console/src/api/materialize/useSubscribe.ts +++ b/console/src/api/materialize/useSubscribe.ts @@ -7,12 +7,11 @@ // the Business Source License, use of this software will be governed // by the Apache License, Version 2.0. -import { PrimitiveAtom, useSetAtom } from "jotai"; +import { PrimitiveAtom, useStore } from "jotai"; import { RawBuilder } from "kysely"; import React from "react"; import { useAutomaticallyConnectSocket } from "~/hooks/useAutomaticallyConnectSocket"; -import { getStore } from "~/jotai"; import { useCurrentEnvironmentHttpAddress } from "~/store/environments"; import { queryBuilder } from "./db"; @@ -25,6 +24,11 @@ import { SubscribeState, UpsertKeyFunction, } from "./SubscribeManager"; +import { + createAtomSubscribeSession, + createCollectionSubscribeSession, + ScopeLoadableAtom, +} from "./subscribeSession"; export { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; @@ -102,14 +106,14 @@ export function useSubscribe>( } /** - * Executes a subscribe query, storing state in the provided atom. - * All updates are reduced to the current set of values at latest closed timestamp. + * Runs an app-session subscribe that reduces into the provided atom, for as + * long as the calling component is mounted. All wiring (connection lifecycle, + * atom writes, region-change reset) lives in createAtomSubscribeSession. * - * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. + * `options` must be referentially stable (declare it at module scope); a new + * object restarts the subscribe session. * - * The `select` and `upsertKey` functions are not expected to be stable, and new - * function instances for these options will not restart the subscribe. On each render, - * the socket will update the function reference. + * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. */ export function useGlobalUpsertSubscribe>( options: UseSubscribeOptions & { @@ -117,54 +121,30 @@ export function useGlobalUpsertSubscribe>( atom: PrimitiveAtom>; }, ) { - const setValue = useSetAtom(options.atom); - const httpAddress = useCurrentEnvironmentHttpAddress(); + const store = useStore(); const request = useSubscribeRequest(options.subscribe); - const [subscribe] = React.useState( - new SubscribeManager({ + React.useEffect(() => { + if (!request) return; + const session = createAtomSubscribeSession({ + store, request, - httpAddress, - upsert: { - key: options.upsertKey, - }, - sessionVariables: { - cluster: options?.clusterName, - }, - closeSocketOnComplete: options?.closeSocketOnComplete, + atom: options.atom, + upsertKey: options.upsertKey, select: options.select, - }), - ); - useAutomaticallyConnectSocket({ - target: subscribe, - subscribe, - request, - }); - - React.useEffect(() => { - const cleanup = subscribe.onChange(() => { - const snapshot = subscribe.getSnapshot(); - const current = getStore().get(options.atom); - if (current === snapshot) return; - - // Hold cached atom data through a fresh manager's empty pre-snapshot state. - const snapshotIsEmptyPreload = - !snapshot.snapshotComplete && !snapshot.data.length && !snapshot.error; - if (snapshotIsEmptyPreload && current.data.length) return; - - setValue(snapshot); + sessionVariables: { cluster: options.clusterName }, + closeSocketOnComplete: options.closeSocketOnComplete, }); - return cleanup; - }, [options.atom, setValue, subscribe]); - - return { - subscribe, - }; + return session.destroy; + }, [store, request, options]); } /** - * Executes a subscribe query and feeds the upsert-reduced state into a TanStack DB - * collection (via createSubscribeCollection). Mirrors useGlobalUpsertSubscribe but - * targets a collection + status atom instead of a single SubscribeState atom. + * Runs an app-session subscribe that feeds a TanStack DB collection, for as + * long as the calling component is mounted. Mirrors useGlobalUpsertSubscribe + * with createCollectionSubscribeSession owning the wiring. + * + * `options` must be referentially stable (declare it at module scope); a new + * object restarts the subscribe session. * * Note that the subscribe statement must have WITH (PROGRESS) and ENVELOPE UPSERT. */ @@ -175,50 +155,25 @@ export function useGlobalSubscribeCollection< options: UseSubscribeOptions & { upsertKey: UpsertKeyFunction; target: SubscribeCollection; + scopeAtom?: ScopeLoadableAtom; }, ) { - const httpAddress = useCurrentEnvironmentHttpAddress(); + const store = useStore(); const request = useSubscribeRequest(options.subscribe); - const target = options.target; - const [subscribe] = React.useState( - new SubscribeManager({ + React.useEffect(() => { + if (!request) return; + const session = createCollectionSubscribeSession({ + store, request, - httpAddress, - upsert: { - key: options.upsertKey, - }, - sessionVariables: { - cluster: options?.clusterName, - }, - closeSocketOnComplete: options?.closeSocketOnComplete, + target: options.target, + scopeAtom: options.scopeAtom, + upsertKey: options.upsertKey, select: options.select, - }), - ); - useAutomaticallyConnectSocket({ - target: subscribe, - subscribe, - request, - }); - - React.useEffect(() => { - // applySnapshot itself drops empty pre-snapshots when state already exists, - // so the manager can push every snapshot unconditionally. - const apply = () => target.applySnapshot(subscribe.getSnapshot()); - const cleanup = subscribe.onChange(apply); - apply(); - return cleanup; - }, [target, subscribe]); - - // Keep the collection actively syncing for the app session by holding a - // subscriber, so it doesn't pause/GC when no component is querying it. - React.useEffect(() => { - const subscription = target.collection.subscribeChanges(() => {}); - return () => subscription.unsubscribe(); - }, [target]); - - return { - subscribe, - }; + sessionVariables: { cluster: options.clusterName }, + closeSocketOnComplete: options.closeSocketOnComplete, + }); + return session.destroy; + }, [store, request, options]); } /** diff --git a/console/src/platform/object-explorer/allNamespacesCollection.ts b/console/src/platform/object-explorer/allNamespacesCollection.ts index c980e80839668..c91590620823c 100644 --- a/console/src/platform/object-explorer/allNamespacesCollection.ts +++ b/console/src/platform/object-explorer/allNamespacesCollection.ts @@ -9,7 +9,7 @@ import { useLiveQuery } from "@tanstack/react-db"; import { useAtomValue } from "jotai"; -import { useEffect, useMemo } from "react"; +import { useMemo } from "react"; import { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; import { @@ -19,7 +19,7 @@ import { import { createSubscribeCollection } from "~/api/materialize/subscribeCollection"; import { SubscribeRow } from "~/api/materialize/SubscribeManager"; import { useGlobalSubscribeCollection } from "~/api/materialize/useSubscribe"; -import { useSyncEngineCacheScope } from "~/store/syncEngineCache"; +import { syncEngineCacheScopeLoadableAtom } from "~/store/syncEngineCache"; /** Subset of AllNamespaceItem stored per row, matching the existing atom's select. */ export type NamespaceItem = Pick< @@ -39,29 +39,23 @@ export const allNamespacesCollection = createSubscribeCollection( }, ); -export function useSubscribeToAllNamespacesCollection() { - const scope = useSyncEngineCacheScope(); - useEffect(() => { - if (scope) allNamespacesCollection.hydrate(scope); - }, [scope]); - - const subscribe = useMemo(() => { - return buildSubscribeQuery(buildAllNamespacesQuery(), { - upsertKey: ["schemaId", "databaseId"], - }); - }, []); +const ALL_NAMESPACES_SUBSCRIBE_OPTIONS = { + target: allNamespacesCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + subscribe: buildSubscribeQuery(buildAllNamespacesQuery(), { + upsertKey: ["schemaId", "databaseId"], + }), + select: (row: SubscribeRow): NamespaceItem => ({ + schemaId: row.data.schemaId, + schemaName: row.data.schemaName, + databaseId: row.data.databaseId, + databaseName: row.data.databaseName, + }), + upsertKey: (row: SubscribeRow) => namespaceKey(row.data), +}; - return useGlobalSubscribeCollection({ - target: allNamespacesCollection, - subscribe, - select: (row: SubscribeRow) => ({ - schemaId: row.data.schemaId, - schemaName: row.data.schemaName, - databaseId: row.data.databaseId, - databaseName: row.data.databaseName, - }), - upsertKey: (row: SubscribeRow) => namespaceKey(row.data), - }); +export function useSubscribeToAllNamespacesCollection() { + useGlobalSubscribeCollection(ALL_NAMESPACES_SUBSCRIBE_OPTIONS); } /** Returns the namespace items for the object explorer tree, from the collection. */ diff --git a/console/src/platform/shell/CommandResult.tsx b/console/src/platform/shell/CommandResult.tsx index 65c065539992d..0ed4bcb76371a 100644 --- a/console/src/platform/shell/CommandResult.tsx +++ b/console/src/platform/shell/CommandResult.tsx @@ -115,15 +115,21 @@ const CommandResult = ({ ? `Returned in ${formatCommandDuration(timeTaken)}` : null; - // If the error comes from a query cancellation, we don't want to display it as - // an error + // A cancelled query and a dropped connection both surface as errors so that + // they get an explanation in the command block, but neither is a failure the + // server reported. Drop the "Error:" prefix for both. const errorOutputPropOverrides = - error && error.code === ErrorCode.QUERY_CANCELED + error && + (error.code === ErrorCode.QUERY_CANCELED || + error.code === ErrorCode.CONNECTION_FAILURE) ? { errorMessageOverride: capitalizeSentence(error.message, false), } : {}; + // NOTE: this deliberately does not include CONNECTION_FAILURE. An + // interrupted command never receives an end timestamp, so `timeTakenStr` is + // null and the element these props apply to is not rendered at all. const codePropOverrides = error?.code === ErrorCode.QUERY_CANCELED ? { diff --git a/console/src/platform/shell/HistoryOutput.test.tsx b/console/src/platform/shell/HistoryOutput.test.tsx new file mode 100644 index 0000000000000..383544a7a3c78 --- /dev/null +++ b/console/src/platform/shell/HistoryOutput.test.tsx @@ -0,0 +1,114 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { screen } from "@testing-library/react"; +import { Setter } from "jotai"; +import mitt from "mitt"; +import React from "react"; + +import { + defaultRegionId, + healthyEnvironment, + renderComponent, + setFakeEnvironment, +} from "~/test/utils"; + +import { COMMAND_INTERRUPTED_MESSAGE } from "./constants"; +import HistoryOutput from "./HistoryOutput"; +import { createInterruptedCommandError } from "./machines/webSocketFsm"; +import { CommandOutput, historyItemAtom, shellStateAtom } from "./store/shell"; +import { + ShellWebsocketContext, + ShellWebsocketContextType, + ShellWSEmitterEvents, +} from "./useShellWebsocket"; + +// The command echo reads Chakra's color mode, which the test providers do not +// install. Nothing here asserts on syntax highlighting. +vi.mock("./SyntaxHighlightedBlock", () => ({ + default: ({ value }: { value: string }) => 
{value}
, +})); + +const HISTORY_ID = "1"; + +const emitter = mitt(); + +const websocketContext: ShellWebsocketContextType = { + send: vi.fn(), + commitToHistory: vi.fn(), + isSocketInitializing: false, + isSocketError: false, + isSocketAvailable: true, + cacheCommand: vi.fn(), + on: emitter.on, + off: emitter.off, +}; + +function commandOutput(overrides: Partial = {}): CommandOutput { + return { + kind: "command", + historyId: HISTORY_ID, + command: "SHOW MATERIALIZED VIEWS;", + statements: [{ query: "SHOW MATERIALIZED VIEWS;" }], + commandSentTimeMs: 0, + commandResults: [{ notices: [] }], + commandResultsDisplayStates: [ + { + isSubscribeManager: false, + isFollowingSubscribeManager: true, + currentTablePage: 0, + currentSubscribeManagerTablePage: 0, + }, + ], + ...overrides, + }; +} + +async function renderHistoryOutput(historyItem: CommandOutput) { + return renderComponent( + + + , + { + initializeState: async ({ set }: { set: Setter }) => { + await setFakeEnvironment(set, defaultRegionId, healthyEnvironment); + // Retry is only enabled once the socket is idle again. + set(shellStateAtom, (prev) => ({ + ...prev, + webSocketState: "readyForQuery", + })); + set(historyItemAtom(HISTORY_ID), historyItem); + }, + }, + ); +} + +describe("HistoryOutput", () => { + it("explains why an interrupted command needs to be retried", async () => { + await renderHistoryOutput( + commandOutput({ + interrupted: true, + error: createInterruptedCommandError(), + }), + ); + + expect(screen.getByText(COMMAND_INTERRUPTED_MESSAGE)).toBeVisible(); + + const retry = screen.getByRole("button", { name: "Retry" }); + expect(retry).toBeVisible(); + expect(retry).toBeEnabled(); + }); + + it("shows neither the message nor Retry for a command that was not interrupted", async () => { + await renderHistoryOutput(commandOutput()); + + expect(screen.queryByText(COMMAND_INTERRUPTED_MESSAGE)).toBeNull(); + expect(screen.queryByRole("button", { name: "Retry" })).toBeNull(); + }); +}); diff --git a/console/src/platform/shell/constants.ts b/console/src/platform/shell/constants.ts index 407c2f606a423..1dd0f58644627 100644 --- a/console/src/platform/shell/constants.ts +++ b/console/src/platform/shell/constants.ts @@ -19,3 +19,9 @@ export const TUTORIAL_WIDTH = "600px"; export const CONNECTION_LOST_NOTICE_MESSAGE = "The connection was interrupted. Some session state may have been lost."; + +// Shown on a command that was in flight when the WebSocket closed. The socket +// has no heartbeat, so a connection that died while idle is only discovered +// when the next command is sent into it. +export const COMMAND_INTERRUPTED_MESSAGE = + "The connection to Materialize was lost. This command may not have run. Press Retry to send it again."; diff --git a/console/src/platform/shell/machines/webSocketFsm.test.ts b/console/src/platform/shell/machines/webSocketFsm.test.ts index 29ab3d2b582d1..e155f339c6e8b 100644 --- a/console/src/platform/shell/machines/webSocketFsm.test.ts +++ b/console/src/platform/shell/machines/webSocketFsm.test.ts @@ -12,7 +12,7 @@ import { interpret } from "@xstate/fsm"; import { ErrorCode, MzDataType } from "~/api/materialize/types"; import { createHistoryId } from "../historyId"; -import webSocketFsm from "./webSocketFsm"; +import webSocketFsm, { createInterruptedCommandError } from "./webSocketFsm"; const MOCKED_HISTORY_ID = "1"; @@ -229,6 +229,55 @@ describe("webSocketFsm", () => { machine.state.context.latestCommandOutput?.commandResults[0].notices[0], ).toEqual(notice); }); + + it("connection closed while a command is in flight marks it interrupted and explains why", () => { + const machine = interpret(webSocketFsm); + machine.start(); + machine.send("READY_FOR_QUERY"); + machine.send({ + type: "SEND", + command: "SELECT 5;", + statements: [{ query: "SELECT 5;" }], + }); + machine.send("COMMAND_STARTING_HAS_ROWS"); + + machine.send({ type: "CONNECTION_CLOSED" }); + + expect(machine.state.matches("interrupted")).toBeTruthy(); + expect( + machine.state.context.latestCommandOutput?.interrupted, + ).toBeTruthy(); + expect(machine.state.context.latestCommandOutput?.error).toEqual( + createInterruptedCommandError(), + ); + }); + + it("connection closed after a server error keeps the server error", () => { + const machine = interpret(webSocketFsm); + machine.start(); + machine.send("READY_FOR_QUERY"); + machine.send({ + type: "SEND", + command: "SELECT nuke_materialize();", + statements: [{ query: "SELECT nuke_materialize();" }], + }); + machine.send("COMMAND_STARTING_HAS_ROWS"); + + const error = { + message: "function nuke_materialize does not exist", + code: ErrorCode.UNDEFINED_FUNCTION, + }; + machine.send({ type: "ERROR", error }); + + // The socket can close before READY_FOR_QUERY arrives, which leaves the + // machine in a state that still counts as processing. + machine.send({ type: "CONNECTION_CLOSED" }); + + expect( + machine.state.context.latestCommandOutput?.interrupted, + ).toBeTruthy(); + expect(machine.state.context.latestCommandOutput?.error).toEqual(error); + }); }); it("should merge a notice into a command that results in an error", () => { diff --git a/console/src/platform/shell/machines/webSocketFsm.ts b/console/src/platform/shell/machines/webSocketFsm.ts index 9a716fbf6eac8..e79b91b7103b9 100644 --- a/console/src/platform/shell/machines/webSocketFsm.ts +++ b/console/src/platform/shell/machines/webSocketFsm.ts @@ -12,11 +12,13 @@ import { assign, createMachine, StateMachine } from "@xstate/fsm"; import { Column, Error, + ErrorCode, ExtendedRequestItem, Notice, } from "~/api/materialize/types"; import { assert } from "~/util"; +import { COMMAND_INTERRUPTED_MESSAGE } from "../constants"; import { CommandOutput, CommandResult, @@ -115,6 +117,19 @@ function getLatestCommandResult(latestCommandOutput?: CommandOutput) { return latestCommandResult; } +/** + * The error a command carries when the connection dropped before it finished. + * + * Materialize never sends 08006 itself, so this code unambiguously marks an + * error the client synthesized rather than one the server reported. + */ +export function createInterruptedCommandError(): Error { + return { + message: COMMAND_INTERRUPTED_MESSAGE, + code: ErrorCode.CONNECTION_FAILURE, + }; +} + const markCommandAsInterrupted = assign< WebSocketFsmContext, ConnectionClosedEvent @@ -122,6 +137,9 @@ const markCommandAsInterrupted = assign< latestCommandOutput: ({ latestCommandOutput }) => { if (latestCommandOutput) { latestCommandOutput.interrupted = true; + if (!latestCommandOutput.error) { + latestCommandOutput.error = createInterruptedCommandError(); + } } return latestCommandOutput; }, diff --git a/console/src/store/allClusters.ts b/console/src/store/allClusters.ts index ec84fbcfa2f0f..d300aebd380d4 100644 --- a/console/src/store/allClusters.ts +++ b/console/src/store/allClusters.ts @@ -14,7 +14,10 @@ import { buildClustersQuery, Cluster, } from "~/api/materialize/cluster/clusterList"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -26,22 +29,18 @@ export const allClusters = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllClusters() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery( - buildClustersQuery({ queryOwnership: false, includeSystemObjects: true }), - { - upsertKey: "id", - }, - ); - }, []); +const ALL_CLUSTERS_SUBSCRIBE_OPTIONS = { + atom: allClusters, + subscribe: buildSubscribeQuery( + buildClustersQuery({ queryOwnership: false, includeSystemObjects: true }), + { upsertKey: "id" }, + ), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allClusters, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllClusters() { + useGlobalUpsertSubscribe(ALL_CLUSTERS_SUBSCRIBE_OPTIONS); } export function useAllClusters() { diff --git a/console/src/store/allObjects.ts b/console/src/store/allObjects.ts index 549e56365a804..6ce6e0a6d66ce 100644 --- a/console/src/store/allObjects.ts +++ b/console/src/store/allObjects.ts @@ -14,7 +14,10 @@ import { buildAllObjectsQuery, DatabaseObject, } from "~/api/materialize/objects"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -26,17 +29,15 @@ export const allObjects = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllObjects() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery(buildAllObjectsQuery(), { upsertKey: "id" }); - }, []); +const ALL_OBJECTS_SUBSCRIBE_OPTIONS = { + atom: allObjects, + subscribe: buildSubscribeQuery(buildAllObjectsQuery(), { upsertKey: "id" }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allObjects, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllObjects() { + useGlobalUpsertSubscribe(ALL_OBJECTS_SUBSCRIBE_OPTIONS); } export function useAllObjects() { diff --git a/console/src/store/allObjectsCollection.ts b/console/src/store/allObjectsCollection.ts index 7f17569771a60..7467ca0ebc4f9 100644 --- a/console/src/store/allObjectsCollection.ts +++ b/console/src/store/allObjectsCollection.ts @@ -8,13 +8,14 @@ // by the Apache License, Version 2.0. import { useLiveQuery } from "@tanstack/react-db"; -import { useAtomValue } from "jotai"; +import { useAtomValue, useStore } from "jotai"; import React from "react"; import { DatabaseObject } from "~/api/materialize/objects"; import { createSubscribeCollection } from "~/api/materialize/subscribeCollection"; +import { createAtomFedCollectionSession } from "~/api/materialize/subscribeSession"; import { allObjects } from "~/store/allObjects"; -import { useSyncEngineCacheScope } from "~/store/syncEngineCache"; +import { syncEngineCacheScopeLoadableAtom } from "~/store/syncEngineCache"; /** * TanStack DB-backed view of the `allObjects` jotai atom, so consumers can run @@ -35,14 +36,16 @@ export const allObjectsCollection = createSubscribeCollection({ * tree is mounted, and sync re-seeds from the retained row set if it restarts. */ export function useSubscribeToAllObjectsCollection() { - const state = useAtomValue(allObjects); - const scope = useSyncEngineCacheScope(); + const store = useStore(); React.useEffect(() => { - if (scope) allObjectsCollection.hydrate(scope); - }, [scope]); - React.useEffect(() => { - allObjectsCollection.applySnapshot(state); - }, [state]); + const session = createAtomFedCollectionSession({ + store, + sourceAtom: allObjects, + target: allObjectsCollection, + scopeAtom: syncEngineCacheScopeLoadableAtom, + }); + return session.destroy; + }, [store]); } /** diff --git a/console/src/store/allRoles.ts b/console/src/store/allRoles.ts index 40fe7971bb90a..d8cf3dc44ea8c 100644 --- a/console/src/store/allRoles.ts +++ b/console/src/store/allRoles.ts @@ -8,7 +8,7 @@ // by the Apache License, Version 2.0. import { atom, useAtomValue } from "jotai"; -import React, { useMemo } from "react"; +import React from "react"; import { buildSubscribeQuery } from "~/api/materialize/buildSubscribeQuery"; import { @@ -29,19 +29,17 @@ export const allRoles = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllRoles() { - const subscribe = useMemo(() => { - return buildSubscribeQuery(buildRolesListQuery(), { - upsertKey: ["roleName"], - }); - }, []); +const ALL_ROLES_SUBSCRIBE_OPTIONS = { + atom: allRoles, + subscribe: buildSubscribeQuery(buildRolesListQuery(), { + upsertKey: ["roleName"], + }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.roleName, +}; - return useGlobalUpsertSubscribe({ - atom: allRoles, - subscribe, - select: (row: SubscribeRow) => row.data, - upsertKey: (row: SubscribeRow) => row.data.roleName, - }); +export function useSubscribeToAllRoles() { + useGlobalUpsertSubscribe(ALL_ROLES_SUBSCRIBE_OPTIONS); } export function useAllRoles() { diff --git a/console/src/store/allSchemas.ts b/console/src/store/allSchemas.ts index b80c7de02fac5..6d57a89f2cc5d 100644 --- a/console/src/store/allSchemas.ts +++ b/console/src/store/allSchemas.ts @@ -15,7 +15,10 @@ import { buildAllSchemaListQuery, SchemaWithOptionalDatabase, } from "~/api/materialize/schemaList"; -import { SubscribeState } from "~/api/materialize/SubscribeManager"; +import { + SubscribeRow, + SubscribeState, +} from "~/api/materialize/SubscribeManager"; import { buildSubscribeQuery, useGlobalUpsertSubscribe, @@ -27,17 +30,17 @@ export const allSchemas = atom>({ snapshotComplete: false, }); -export function useSubscribeToAllSchemas() { - const subscribe = React.useMemo(() => { - return buildSubscribeQuery(buildAllSchemaListQuery(), { upsertKey: "id" }); - }, []); +const ALL_SCHEMAS_SUBSCRIBE_OPTIONS = { + atom: allSchemas, + subscribe: buildSubscribeQuery(buildAllSchemaListQuery(), { + upsertKey: "id", + }), + select: (row: SubscribeRow) => row.data, + upsertKey: (row: SubscribeRow) => row.data.id, +}; - return useGlobalUpsertSubscribe({ - atom: allSchemas, - subscribe, - select: (row) => row.data, - upsertKey: (row) => row.data.id, - }); +export function useSubscribeToAllSchemas() { + useGlobalUpsertSubscribe(ALL_SCHEMAS_SUBSCRIBE_OPTIONS); } export function useAllSchemas(options?: { includeSystemSchemas?: boolean }) { diff --git a/console/src/store/syncEngineCache.ts b/console/src/store/syncEngineCache.ts index e098efa2d1758..acd211f9105e0 100644 --- a/console/src/store/syncEngineCache.ts +++ b/console/src/store/syncEngineCache.ts @@ -7,26 +7,41 @@ // the Business Source License, use of this software will be governed // by the Apache License, Version 2.0. -import { useAtomValue } from "jotai"; +import { atom } from "jotai"; +import { loadable } from "jotai/utils"; -import { useMaybeCurrentOrganizationId } from "~/api/auth"; +import { + fetchCurrentOrganization, + isOrganizationFetchEnabled, + queryKeys as authQueryKeys, +} from "~/api/auth"; +import { Organization } from "~/api/cloudGlobalApi"; +import { appConfigAtom } from "~/config/store"; +import { getQueryClient } from "~/queryClient"; import { currentRegionIdSyncAtom } from "~/store/environments"; /** * The auth/region scope for the sync-engine localStorage caches, as - * `organizationId|regionId`. Returns undefined until both are known, so callers - * defer seeding until the scope is settled and never share one tenant's cache - * with another on the shared `console.materialize.com` origin. + * `organizationId|regionId`. Resolves to undefined until both are known, so + * consumers defer cache seeding until the scope is settled and never share one + * tenant's cache with another on a shared origin. */ -export function useSyncEngineCacheScope(): string | undefined { - const maybeOrganizationId = useMaybeCurrentOrganizationId(); - const regionId = useAtomValue(currentRegionIdSyncAtom); +export const syncEngineCacheScopeAtom = atom(async (get) => { + const regionId = get(currentRegionIdSyncAtom); + if (!regionId) return undefined; - const organizationIdLoading = - maybeOrganizationId !== null && maybeOrganizationId.isLoading; - if (organizationIdLoading || !regionId) return undefined; - - const organizationId = - maybeOrganizationId !== null ? maybeOrganizationId.data : undefined; + const organizationId = isOrganizationFetchEnabled(get(appConfigAtom)) + ? ( + await getQueryClient().ensureQueryData({ + queryKey: authQueryKeys.currentOrganization(), + queryFn: fetchCurrentOrganization, + }) + ).id + : undefined; return `${organizationId ?? ""}|${regionId}`; -} +}); + +/** Synchronous view of the scope for store.sub consumers (subscribe sessions). */ +export const syncEngineCacheScopeLoadableAtom = loadable( + syncEngineCacheScopeAtom, +); diff --git a/doc/user/CLAUDE.md b/doc/user/CLAUDE.md index b818c4fffabd8..0c46c7474d1b4 100644 --- a/doc/user/CLAUDE.md +++ b/doc/user/CLAUDE.md @@ -137,6 +137,14 @@ with two dead sections: - **Symlinked content**: `content/` contains no symlinks today (only `sql-grammar/keywords.txt` is one). +## Verifying docs + +- Test documentation by running its SQL against the Materialize emulator (the + `materialize/materialized` Docker image) before committing, rather than + assuming a query works. +- When writing syntax demos, include both the query and its output, and confirm + the syntax and shown output match what the emulator actually returns. + ## Reviewing docs changes The `docs-content-reviewer` agent and `.prompts/docs_pre_pr_review.md` encode the diff --git a/doc/user/content/concepts/clusters.md b/doc/user/content/concepts/clusters.md index bbb8f0af79237..cfa8cbd9b533d 100644 --- a/doc/user/content/concepts/clusters.md +++ b/doc/user/content/concepts/clusters.md @@ -16,6 +16,21 @@ aliases: Clusters are pools of compute resources (CPU, memory, and scratch disk space) for running your workloads. +## Resource isolation + +Clusters provide **resource isolation.** Each cluster provisions dedicated +compute resources and can fail independently from other clusters. All workloads +on a given cluster compete for access to that cluster's compute resources. + +Workloads on different clusters are strictly isolated from one another. That is, +a given workload has access only to the CPU, memory, and scratch disk of the +cluster it runs on. + +Resource isolation lets you place workloads on separate clusters to prevent +them from competing for compute resources. + +See also [three-tier architecture](#three-tier-architecture-in-production). + ## Clusters and workloads The following operations require a cluster in Materialize: @@ -37,7 +52,8 @@ SET CLUSTER = 'my_transform_cluster'; [`SELECT`] and [`SUBSCRIBE`] statements run in the session's active cluster. Objects that require compute (e.g., indexes, materialized views, sources) are -associated with a cluster when they are created, either: +associated with a cluster when they are created. The associated cluster is +either: - the session's active cluster by default, or @@ -54,22 +70,6 @@ associated with a cluster when they are created, either: For more on indexes and clusters, see [Indexes](/concepts/indexes/). -## Resource isolation - -Clusters provide **resource isolation.** Each cluster provisions dedicated -compute resources and can fail independently from other clusters. All workloads -on a given cluster compete for access to that cluster's compute resources. - -Workloads on different clusters are strictly isolated from one another. That is, -a given workload has access only to the CPU, memory, and scratch disk of the -cluster it runs on. - -Resource isolation lets you place workloads on separate clusters to prevent -them from competing for compute resources: for example, sources in one -cluster, materialized views in a second, and indexes that serve queries in a -third, as in the recommended [three-tier -architecture](#three-tier-architecture-in-production). - ## Cluster replicas The [replication factor](/sql/create-cluster/#replication-factor) of a cluster @@ -103,6 +103,146 @@ When provisioning replicas, See also [Hydration considerations](#hydration-considerations). +## Lifecycle of a cluster + +Whenever a cluster starts running a workload (after you create it, resize it, +or one of its replicas restarts), its replicas move through a sequence of states +before results are fully up to date. Knowing which state a cluster is in tells +you whether it is making progress or is stuck. + +The queries below monitor a cluster named `lifecycle_demo` that hosts the +materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both +built on a continuously-updating `AUCTION` load-generator source. Substitute +your own cluster and object names. + +### Provisioning + +Replicas are scheduled and brought online. A cluster with a [replication +factor](#cluster-replicas) of `0` has no compute and never leaves this state. To +monitor progress, check that replicas report `online` in +[`mz_cluster_replica_statuses`](/reference/system-catalog/mz_internal/#mz_cluster_replica_statuses), +and confirm the cluster has replicas via +[`mz_clusters`](/reference/system-catalog/mz_catalog/#mz_clusters). + +```mzsql +SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason +FROM mz_internal.mz_cluster_replica_statuses st +JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id +JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id +WHERE c.name = 'lifecycle_demo' +ORDER BY r.name; +``` + +```none + cluster | replica | size | status | reason +----------------+---------+------+--------+-------- + lifecycle_demo | r1 | 25cc | online | +(1 row) +``` + +The `reason` column is empty while the replica is `online`, and reports why a +replica is unavailable otherwise. + +### Hydrating + +Each replica reconstructs its in-memory state by reading from Materialize's +storage layer (see [hydration](/concepts/hydration/)). While an object is +hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To +monitor progress, check the `hydrated` flag per object in +[`mz_hydration_statuses`](/reference/system-catalog/mz_internal/#mz_hydration_statuses), +where the `replica_id` stays blank until the object attaches to a replica. For +indexes and materialized views, +[`mz_compute_hydration_statuses`](/reference/system-catalog/mz_internal/#mz_compute_hydration_statuses) +also reports how long hydration took. + +```mzsql +SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time +FROM mz_internal.mz_compute_hydration_statuses ch +JOIN mz_objects o ON o.id = ch.object_id +JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id +WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') +ORDER BY o.name; +``` + +```none + object | type | replica | hydrated | hydration_time +---------------------+-------------------+---------+----------+----------------- + bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 + bids_by_auction_idx | index | r1 | t | 00:00:00.000019 + bids_load | materialized-view | r1 | t | 00:00:05.6032 +(3 rows) +``` + +The light view and index hydrate in microseconds, while the larger `bids_load` +view takes about 5.6 seconds. A larger object with more state to reconstruct +shows a longer, more visible hydration window. + +### Catching up + +Once hydrated, the cluster processes the backlog of input updates that +accumulated while it was unavailable, so its total lag starts high and comes +down. To monitor progress, watch `lag` decrease in +[`mz_wallclock_global_lag_recent_history`](/reference/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), +or break the lag down by input with +[`mz_materialization_lag`](/reference/system-catalog/mz_internal/#mz_materialization_lag). + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name IN ('bids_by_auction', 'bids_load') +ORDER BY o.name; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+------------------+------------------+---------------------+---------------------- + bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids + bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids +``` + +Both objects trail their slowest input, the `bids` source, by tens of seconds. +As the cluster works through the backlog, these lags fall. + +### Steady state + +The cluster has caught up and its lag holds low and roughly constant, typically +a few seconds. Re-running the lag query confirms the objects have caught up to +their input. + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name = 'bids_by_auction'; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+-----------+------------+---------------------+---------------------- + bids_by_auction | 00:00:00 | 00:00:00 | bids | bids +(1 row) +``` + +Wallclock lag in +[`mz_wallclock_global_lag_recent_history`](/reference/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) +holds near-constant at a few seconds. A lag that instead climbs steadily, at +about one minute per minute, means the cluster has stopped making progress. + +{{< note >}} +Sources go through an additional +[snapshotting](/concepts/snapshotting/) step the first time they run, reading the +initial state of the upstream system before the states above apply. See +[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to +diagnose a cluster that is not progressing through these states. +{{< /note >}} + ## Cluster sizing diff --git a/doc/user/content/concepts/indexes.md b/doc/user/content/concepts/indexes.md index 9d2b69cde128a..286ba55b7b923 100644 --- a/doc/user/content/concepts/indexes.md +++ b/doc/user/content/concepts/indexes.md @@ -12,15 +12,25 @@ aliases: - /self-managed/v25.2/concepts/indexes/ --- -In Materialize, you can create indexes on [views](/concepts/views/#views) and -[materialized views](/concepts/views/#materialized-views) as well as tables, -[sources](/concepts/sources/), and subsources. - ## Overview {{% include-from-yaml data="index_details" name="definition" %}} -## Indexes on sources, tables, and subsources +## Creating indexes on objects + +In Materialize, you can create indexes on [views](/concepts/views/#views) and +[materialized views](/concepts/views/#materialized-views) as well as on +[sources, tables, and subsources](/concepts/sources/). + +To create indexes on an object, use the [`CREATE INDEX`](/sql/create-index/) +command. To create the index in a cluster other than the active cluster, include +the `IN CLUSTER` clause in the `CREATE INDEX` statement. + +{{% include-example file="examples/create_index" example="syntax" %}} + +See [`CREATE INDEX`](/sql/create-index/) for the syntax details. + +### Indexes on sources, tables, and subsources {{< note >}} @@ -29,132 +39,141 @@ or subsources without performing some transformation using a view, etc. {{}} -In Materialize, you can create indexes on a [source and its tables or -subsources](/concepts/sources/) to maintain in-memory up-to-date data within the -cluster you create the index. This can help improve [query -performance](#indexes-and-query-optimizations) such as when [using +In Materialize, you can create indexes on [sources, tables, or +subsources](/concepts/sources/) to maintain up-to-date data in the memory of +the cluster where you create the index. This can help improve [query +performance](#indexes-and-query-optimizations), for example when [using joins](/transform-data/optimization/#join) in your transformation. However, in practice, you may find that you rarely need to index these objects directly. ```mzsql -CREATE INDEX idx_on_my_source_table ON my_source_table (...); +CREATE INDEX idx_on_my_source_table ON my_source_table(...); ``` -## Indexes on views +### Indexes on views -In Materialize, you can create indexes on a [view](/concepts/views/#views "query -saved under a name") to maintain **up-to-date view results in memory** within -the [cluster](/concepts/clusters/) you create the index. +In Materialize, you can [create indexes](/sql/create-index/) on a +[view](/concepts/views/#views "query saved under a name") to maintain +**up-to-date view results in memory** within the [cluster](/concepts/clusters/) +where you create the index. -```mzsql -CREATE INDEX idx_on_my_view ON my_view_name(...) ; -``` +- To create the index in the current active cluster (you can use the `SET + CLUSTER` command to change the active cluster): + + ```mzsql + CREATE INDEX idx_on_my_view ON my_view_name(...); + ``` -During the index creation on a [view](/concepts/views/#views "query saved under -a name"), the view is executed and the view results are stored in memory within -the cluster. **As new data arrives**, the index **incrementally updates** the -view results in memory. +- To create the index in a specified cluster: -Within the cluster, querying an indexed view is **fast** because the results are -already computed and are served from memory. + ```mzsql + CREATE INDEX idx_on_my_view IN CLUSTER serving_cluster ON my_view_name(...); + ``` -For best practices on using indexes, and understanding when to use indexed views -vs. materialized views, see [Usage patterns](#usage-patterns). +During the index creation, the view is executed and the view results are stored +in memory within the cluster. **As new data arrives**, the index **incrementally +updates** the view results in memory. -## Indexes on materialized views +Querying a view from a cluster where the view is indexed is **fast** because +the results are already computed and are served from memory. Querying a view +from a cluster where the view isn't indexed requires executing the view each +time you query it. + +### Indexes on materialized views In Materialize, materialized view results are stored in durable storage and -**incrementally updated** as new data arrives. Indexing a materialized view -makes the already up-to-date view results available **in memory** within the -[cluster](/concepts/clusters/) you create the index. That is, indexes on -materialized views require no additional computation to keep results up-to-date. +**incrementally updated** as new data arrives. [Indexing](/sql/create-index/) a +materialized view makes the already up-to-date view results available **in +memory** within the [cluster](/concepts/clusters/) where you create the index. +That is, indexes on materialized views require no additional computation to keep +results up-to-date. {{< note >}} A materialized view can be queried from any cluster whereas its indexed results -are available only within the cluster you create the index. Querying a -materialized view, whether indexed or not, from any cluster is fast since the -results are already computed. However, querying an indexed materialized view -within the cluster where the index is created is faster since the results are -served from memory rather than from storage. +are available only within the cluster where you create the index. Querying a +materialized view from any cluster, whether the materialized view is indexed or +not, is fast because the results are already computed. However, querying an +indexed materialized view from a cluster where the materialized view is indexed +is faster since the results are served from memory rather than from storage. {{}} -For best practices on using indexes, and understanding when to use indexed views -vs. materialized views, see [Usage patterns](#usage-patterns). +- To create the index in the current active cluster (you can use the `SET + CLUSTER` command to change the active cluster): -```mzsql -CREATE INDEX idx_on_my_mat_view ON my_mat_view_name(...) ; -``` + ```mzsql + CREATE INDEX idx_on_my_mat_view ON my_mat_view_name(...); + ``` -## Indexes and clusters +- To create the index in a specified cluster: -{{% include-from-yaml data="index_details" name="index-cluster-local" %}} + ```mzsql + CREATE INDEX idx_on_my_mat_view IN CLUSTER serving_cluster ON my_mat_view_name(...); + ``` -As such, queries issued from a different cluster cannot use the index. +## Properties -For example, to create an index in the current cluster: +### Cluster-local -```mzsql -CREATE INDEX idx_on_my_view ON my_view_name(...) ; -``` +{{% include-from-yaml data="index_details" name="index-cluster-local" %}} As +such, references to the indexed object from a different cluster cannot use the +index. -You can also explicitly specify the cluster: +### Data distribution and ordering -```mzsql -CREATE INDEX idx_on_my_view IN CLUSTER active_cluster ON my_view (...); -``` +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} -## Usage patterns +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} -### Index usage +### Serving ad-hoc queries -{{% important %}} -Indexes are local to a cluster. Queries in one cluster cannot use the indexes in another, different cluster. -{{% /important %}} +Within a cluster, all ad-hoc queries that reference an indexed object read from +the index, regardless of whether the index is optimized for the query. This +includes queries that do not specify a `WHERE` condition on the index key. +Because the indexed results are already up-to-date and in memory, reading from +an index avoids recomputing the results. -Unlike some other databases, Materialize can use an index to serve query results -even if the query does not specify a `WHERE` condition on the index key. Serving -queries from an index is fast since the results are already up-to-date and in -memory. +- **Point lookups**: For queries that specify an equality condition on the full + index key, Materialize can perform a point lookup, reading only the matching + records from the index. Point lookups are the most efficient use of an index. + See [Point lookups](#point-lookups) for the exact requirements. -For example, consider the following index: +- **Index scans**: Otherwise, Materialize scans the index. Although the indexed + results are already up-to-date and in memory, a full index scan must examine + the indexed results and is less efficient than a point lookup. The performance + of full index scans degrades with data volume. -```mzsql -CREATE INDEX idx_orders_view_qty ON orders_view (quantity); -``` +### Index use by objects -Materialize will maintain the `orders_view` in memory in `idx_orders_view_qty`, -and it will be able to use the index to serve a various queries on the -`orders_view` (and not just queries that specify conditions on -`orders_view.quantity`). +{{% include-from-yaml data="index_details" name="index-reuse" %}} -Materialize can use the index for the following queries (issued from the same -cluster as the index) on `orders_view`: +To inspect index reuse and dependencies: -```mzsql -SELECT * FROM orders_view; -- scans the index -SELECT * FROM orders_view WHERE status = 'shipped'; -- scans the index -SELECT * FROM orders_view WHERE quantity = 10; -- point lookup on the index -``` +- To check whether a new index would reuse an existing index before creating + it, use [`EXPLAIN CREATE INDEX`](/sql/explain-plan/). + +- To find which indexes and materialized views use an index, query + [`mz_internal.mz_materialization_dependencies`](/reference/system-catalog/mz_internal/#mz_materialization_dependencies). -For the queries that do not specify a condition on the indexed field, -Materialize scans the index. For the query that specifies an equality condition -on the indexed field, Materialize performs a **point lookup** on the index -(i.e., reads just the matching records from the index). Point lookups are the -most efficient use of an index. +### Limitations -#### Point lookups +{{% include-from-yaml data="index_details" name="index-not-optimized" %}} -Materialize performs **point lookup** (i.e., reads just the matching records -from the index) on the index if the query's `WHERE` clause: +## Point lookups vs index scans + +### Point lookups + +Point lookups read just the matching records from the index and are the most +efficient use of an index. Materialize performs a point lookup if the query's +`WHERE` clause: - Specifies equality (`=` or `IN`) condition and **only** equality conditions on **all** the indexed fields. The equality conditions must specify the **exact** index key expression (including type) for point lookups. For example: - If the index is on `round(quantity)`, the query must specify equality - condition on `round(quantity)` (and not just `quanity`) for Materialize to + condition on `round(quantity)` (and not just `quantity`) for Materialize to perform a point lookup. - If the index is on `quantity * price`, the query must specify equality @@ -166,39 +185,55 @@ from the index) on the index if the query's `WHERE` clause: - Only uses `AND` (conjunction) to combine conditions for **different** fields. -Point lookups are the most efficient use of an index. - For queries whose `WHERE` clause meets the point lookup criteria and includes conditions on additional fields (also using `AND` conjunction), Materialize performs a point lookup on the index keys and then filters the results using the additional conditions on the non-indexed fields. -For queries that do not meet the point lookup criteria, Materialize performs a -full index scan (including for range queries). That is, Materialize performs a -full index scan if the `WHERE` clause: +### Index scans + +For queries that do not meet the [point lookup criteria](#point-lookups), +Materialize performs a full index scan (including for range queries). That is, +Materialize performs a full index scan if the `WHERE` clause: - Does not specify **all** the indexed fields. - Does not specify only equality conditions on the index fields or specifies an equality condition that specifies a different value type than the index key type. -- Uses OR (disjunction) to combine conditions for **different** fields. +- Uses `OR` (disjunction) to combine conditions for **different** fields. -Full index scans are less efficient than point lookups. The performance of full +Full index scans are less efficient than point lookups. The performance of full index scans will degrade with data volume; i.e., as you get more data, full scans will get slower. -#### Examples +### Examples + +Within a cluster, indexes can serve queries that reference an indexed object, +regardless of whether the index is optimized for the query. -Consider again the following index on a view: +Consider the following index on the `orders_view`: ```mzsql -CREATE INDEX idx_orders_view_qty on orders_view (quantity); +CREATE INDEX idx_orders_view_qty ON orders_view (quantity); +``` + +Materialize can use the index to serve various queries on the `orders_view` +(and not just queries that specify conditions on `orders_view.quantity`). For +example: + +```mzsql +SELECT * FROM orders_view; -- scans the index +SELECT * FROM orders_view WHERE status = 'shipped'; -- scans the index +SELECT * FROM orders_view WHERE quantity = 10; -- point lookup on the index ``` +For the queries that do not satisfy the [point-lookup +conditions](#point-lookups), Materialize scans the index. + The following table shows various queries and whether Materialize performs a point lookup or an index scan. -{{< yaml-table data="examples/index_usage/index_usage_key_quantity" >}} +{{% yaml-table data="examples/index_usage/index_usage_key_quantity" %}} Consider that the view has an index on the `quantity` and `price` fields instead of an index on the `quantity` field: @@ -208,11 +243,9 @@ DROP INDEX idx_orders_view_qty; CREATE INDEX idx_orders_view_qty_price on orders_view (quantity, price); ``` -{{< yaml-table data="examples/index_usage/index_usage_key_quantity_price" >}} - -#### Limitations +{{% yaml-table data="examples/index_usage/index_usage_key_quantity_price" %}} -{{% include-headless "/headless/index-ordering" %}} +## Usage ### Indexes on views vs. materialized views diff --git a/doc/user/content/headless/index-ordering.md b/doc/user/content/headless/index-ordering.md deleted file mode 100644 index 2489a2afee243..0000000000000 --- a/doc/user/content/headless/index-ordering.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -headless: true ---- - -Indexes in Materialize do not order their keys using the data type's natural -ordering and instead orders by its internal representation of the key (the tuple -of key length and value). - -As such, indexes in Materialize currently do not provide optimizations for: - -- Range queries; that is queries using `>`, `>=`, - `<`, `<=`, `BETWEEN` clauses (e.g., `WHERE - quantity > 10`, `price >= 10 AND price <= 50`, and `WHERE quantity - BETWEEN 10 AND 20`). - -- `GROUP BY`, `ORDER BY` and `LIMIT` clauses. diff --git a/doc/user/content/reference/system-catalog/mz_internal.md b/doc/user/content/reference/system-catalog/mz_internal.md index 76d7e78b76ae4..40165fecba5cf 100644 --- a/doc/user/content/reference/system-catalog/mz_internal.md +++ b/doc/user/content/reference/system-catalog/mz_internal.md @@ -649,8 +649,8 @@ has no replicas — needs operator action." ## `mz_object_dependencies` -The `mz_object_dependencies` table describes the dependency structure between -all database objects in the system. +The `mz_object_dependencies` materialized view describes the dependency +structure between all database objects in the system. | Field | Type | Meaning | @@ -1623,6 +1623,7 @@ The `mz_webhook_sources` table contains a row for each webhook source in the sys + diff --git a/doc/user/content/releases/_index.md b/doc/user/content/releases/_index.md index ad4bc15367714..ab52ca41c6f91 100644 --- a/doc/user/content/releases/_index.md +++ b/doc/user/content/releases/_index.md @@ -19,6 +19,76 @@ Starting with the v26.1.0 release, Materialize releases on a weekly schedule for both Cloud and Self-Managed. See [Release schedule](/releases/schedule) for details. {{}} +## v26.40.0 +*Released to Materialize Cloud: 2026-09-02*
+*Released to Materialize Self-Managed: 2026-09-03*
+ +### Iceberg support for Databricks on AWS {#v26.40-iceberg-support-for-databricks-on-aws} + +{{< public-preview />}} + +Iceberg sinks can now write to Apache Iceberg tables registered in [Databricks +Unity Catalog](/serve-results/sink/iceberg-databricks/) on AWS, reached through +Unity Catalog's Iceberg REST catalog endpoint. Two new Iceberg catalog +connection options make this work: `OAUTH2 SERVER URL`, which names a token +endpoint that does not sit under the catalog URI, and `ACCESS DELEGATION = +'vended-credentials'`, which asks the catalog for temporary, table-scoped +storage credentials instead of static keys. Materialize refreshes both the +OAuth2 token and the vended credentials while the sink runs, so a sink that +outlives one credential lifetime keeps writing. + +```mzsql +CREATE SECRET databricks_oauth + AS ':'; + +CREATE CONNECTION iceberg_catalog_connection TO ICEBERG CATALOG ( + CATALOG TYPE = 'rest', + URL = 'https://.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest', + WAREHOUSE = '', + CREDENTIAL = SECRET databricks_oauth, + OAUTH2 SERVER URL = 'https://.cloud.databricks.com/oidc/v1/token', + SCOPE = 'all-apis', + ACCESS DELEGATION = 'vended-credentials' +); + +CREATE SINK + IN CLUSTER + FROM + INTO ICEBERG CATALOG CONNECTION iceberg_catalog_connection ( + NAMESPACE = '', + TABLE = '' + ) + MODE APPEND + WITH (COMMIT INTERVAL = ''); +``` + +For more information, see: +- [Guide: Databricks on AWS](/serve-results/sink/iceberg-databricks/) +- [`CREATE CONNECTION`: Iceberg catalog](/sql/create-connection/#iceberg-catalog), including [storage access delegation](/sql/create-connection/#iceberg-catalog-access-delegation) +- [`CREATE SINK`: Iceberg](/sql/create-sink/iceberg/), including [append mode](/sql/create-sink/iceberg/#append-mode) + +### Improvements {#v26.40-improvements} +- **Query History for Self-Managed**: Self-Managed deployments can now use the Console's Query History view to debug latency and performance. +- **Bounded staleness isolation is generally available**: The [`bounded staleness `](/reference/isolation-level/#bounded-staleness) transaction [isolation level](/reference/isolation-level/) is out of public preview and is now a supported part of the [`transaction_isolation`](/reference/isolation-level/#setting-isolation-level) surface. See [when to use bounded staleness](/reference/isolation-level/#when-to-use-bounded-staleness) and its [restrictions](/reference/isolation-level/#restrictions). +- **Replica resource usage introspection**: A new `mz_introspection.mz_cluster_replica_resource_usage` relation reports each replica process's own memory, swap, and disk observations at a higher cadence than the roughly once-a-minute orchestrator samples, so a spike between two samples is no longer invisible. + +### Agent Skills {#v26.40-agent-skills} +- **mz-optimize-memory**: A new skill that works top-down from a cluster's largest arrangements to a table of memory-reducing fixes — index changes, outer-join and subquery rewrites, window-function patterns, and arrangement size hints — with rules for estimating each saving before making the change and verifying it afterwards. + +### Bug Fixes {#v26.40-bug-fixes} +- Fixed the MCP servers rejecting `tools/list`, `ping`, and `notifications/initialized` requests that carry any `params` — including the `_meta` field MCP clients are allowed to attach — with a non-JSON-RPC HTTP 422 that clients read as the server being unavailable. +- Fixed `ALTER MATERIALIZED VIEW ... APPLY REPLACEMENT` leaving a stale cached query plan behind, which put `environmentd` into a crash loop on every subsequent restart once a dependency of the replaced definition was dropped, and could otherwise leave the view computing its old definition. +- `CREATE TABLE ... FROM SOURCE` now requires `SELECT` on the source and `USAGE` on its schema, closing a case where `CREATE` on any schema a role controlled was enough to read a source that role had been denied; deployments where a platform team owns sources and application teams attach tables into their own schemas will need those `SELECT` grants added. +- Fixed `ALTER CONNECTION` letting a connection owner keep secrets and connections they lack `USAGE` privileges on, and read those secrets during content checks or connection validation. +- Fixed `ANY`/`ALL` over a `NULL` array or list returning the empty-set answer instead of `NULL`, and `array_position` failing to find `NULL` elements, both now matching PostgreSQL. +- Fixed `round(numeric, scale)` erroring on a scale that reaches past the value's fractional digits, such as `round(123::numeric, 38)`, which could also let filter pushdown discard data a query matched. +- Fixed identity-provider group sync silently finding nothing when `oidc_group_claim` names a claim the token declares directly, such as `roles`, so group-to-role mapping now works without a custom claim or JWT prehook. +- Fixed Arrow ADBC clients failing to connect by adding the missing `pg_type.typsend` column, and fixed `typreceive` rendering as a numeric OID rather than the function name, so those clients resolve every column to its real type. +- Fixed Iceberg sinks ignoring a table's `write.data.path` property and always writing data files to the default location. +- Fixed sources on the Console's Objects page being described in replica-hydration terms, which could show a healthy source as `Not Hydrated`, leave a webhook source with no status, and hold a completed snapshot at 99%. +- Fixed several `dbt-materialize` error paths reporting confusing failures instead of the intended messages, covering the index config parser, renaming a view, dropping an unsupported relation type, connection option parsing, and the version check against a server that is not Materialize. +- Fixed the `mz` CLI failing on read-only commands such as `mz sql` when `mz.toml` sits on a read-only mount. + ## v26.39.0 *Released to Materialize Cloud: 2026-08-26*
*Released to Materialize Self-Managed: 2026-08-27*
diff --git a/doc/user/content/transform-data/faq.md b/doc/user/content/transform-data/faq.md index 4b921c6b0ddb8..755f921b9c261 100644 --- a/doc/user/content/transform-data/faq.md +++ b/doc/user/content/transform-data/faq.md @@ -11,14 +11,28 @@ aliases: - /self-managed/v25.2/transform-data/faq/ --- -## Do indexes in Materialize support `ORDER BY`? +## Are indexes in Materialize optimized for `ORDER BY`? -No. Indexes in Materialize do not support `ORDER BY` clauses. +No. -{{% include-headless "/headless/index-ordering" %}} +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} -## Do indexes in Materialize support range queries? +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} -No. Indexes in Materialize do not support range queries. +As such, Materialize indexes are not optimized for ordered access, including +`ORDER BY` clauses. -{{% include-headless "/headless/index-ordering" %}} +## Are indexes in Materialize optimized for range queries? + +No. + +{{% include-from-yaml data="index_details" name="index-key-distribution" %}} + +{{% include-from-yaml data="index_details" name="index-key-ordering-within-workers" %}} + +As such, Materialize indexes are not optimized for ordered access, including +range queries. + +## Are indexes in Materialize optimized for `GROUP BY` aggregations? + +No. {{< include-from-yaml data="index_details" name="index-groupby-not-optimized" >}} diff --git a/doc/user/content/transform-data/optimization.md b/doc/user/content/transform-data/optimization.md index 0063cfb439f2d..2d7c76f87fdf0 100644 --- a/doc/user/content/transform-data/optimization.md +++ b/doc/user/content/transform-data/optimization.md @@ -448,7 +448,7 @@ The following are the possible index usage types: ### Limitations -{{% include-headless "/headless/index-ordering" %}} +{{% include-from-yaml data="index_details" name="index-not-optimized" %}} ## Query hints diff --git a/doc/user/data/index_details.yml b/doc/user/data/index_details.yml index a5fd97610f84c..bad35ecfa3ca5 100644 --- a/doc/user/data/index_details.yml +++ b/doc/user/data/index_details.yml @@ -6,9 +6,12 @@ - name: definition content: | Materialize indexes maintain the full result set of the indexed object in - the memory of the [cluster](/concepts/clusters/) where the index is - created.[^db-term] The indexed results are kept up-to-date as new data - arrives. + the memory of the [cluster](/concepts/clusters/) where the index is created. + The cluster's workers keep the indexed results up-to-date as new data + arrives. Like clustered[^db-term] hash indexes, Materialize indexes store + the indexed results themselves and are efficient for equality lookups on the + full index key. Materialize indexes are not themselves hash indexes; hashing + is used only to distribute the index across the cluster's workers. ![Materialize index maintains the full result set in memory](/images/indexes/index_in_memory.svg) @@ -17,14 +20,14 @@ ![Materialize indexes do not use a key-pointer structure.](/images/indexes/index_not_key_pointer.svg) - [^db-term]: Materialize indexes are like clustered hash indexes. The term - *clustered index* is a database term unrelated to Materialize clusters, - which are compute resources. + [^db-term]: The term *clustered + index* is a database term unrelated to Materialize clusters, which are + compute resources. # Shared with the clusters concept page, which composes it after the # cross-cluster-accessibility entry in cluster_details.yml. Renders on both # the clusters and indexes pages, so avoid page-root links to either page; -# deep anchor links (e.g., #resource-isolation) are fine — on clusters.md +# deep anchor links (e.g., #resource-isolation) are fine. On clusters.md # they act as same-page jumps. - name: index-cluster-local content: | @@ -32,3 +35,88 @@ in the memory of the cluster where the index is created, and a [cluster's memory](/concepts/clusters/#resource-isolation) cannot be accessed from another cluster. + +# Creation-order semantics for index reuse. The hydration page's "index-order" +# strategy bullet (concepts/hydration.md) states the same create-before- +# consumers rule; keep them consistent. +- name: index-reuse + content: | + Within a cluster, an index can be used not only by ad-hoc queries but also + by other indexes and materialized views. For an index or materialized view + to use another index, however, that index must exist when the dependent + object is created. That is: + + - When you create an index or a materialized view, Materialize plans how + to compute its results at creation time. As part of planning, Materialize + checks whether it can reuse an **existing** index in the **same** + cluster. + + - Because the plan is bound at creation time, creation order matters. An + index or materialized view that is already running will **not** adopt an + index created afterward. To have an existing index or materialized view + use a newer index, drop and recreate the existing object. However, + recreating an index or a materialized view triggers + [hydration](/concepts/hydration/#when-hydration-occurs). + + Ad-hoc queries, by contrast, are planned at query time. They can use any + index that exists in the cluster when the query runs. + + {{< note >}} + Reusing an index saves computation since the dependent objects read the + index's maintained results instead of recomputing them from the base data. + However, each new index has costs related to cluster memory and ongoing + maintenance, especially indexes on regular views. + {{< /note >}} + +- name: index-key-distribution + content: | + The index data is distributed across the cluster's + workers by a hash of the key, which spreads the maintenance and lookup work + across the cluster. + +- name: index-key-ordering-within-workers + content: | + Within each worker, index keys are ordered by their internal representation + (the encoded key's length, then its bytes), not by the data types' natural + ordering. + +# GROUP BY gets no index-specific optimization: the reduce operator always +# builds and maintains its own arrangement keyed by the grouping key, so an +# index on the grouping key does not reduce aggregation work. Single +# paragraph (>-): nested via the {{< >}} angle form inside the GROUP BY +# bullet of the index-not-optimized entry below, and rendered inline after +# "No." on transform-data/faq.md. +- name: index-groupby-not-optimized + content: >- + An index on the grouping key does not reduce the work of computing the + aggregation: Materialize reads the full index and maintains the + aggregation separately. + +# Consumers: concepts/indexes.md and transform-data/optimization.md, both +# under a "Limitations" heading. On optimization.md the list renders with no +# supporting context, so each bullet must stay self-contained and carry its +# own reason where one is needed. The GROUP BY bullet nests the +# index-groupby-not-optimized entry above; it must use the {{< >}} angle +# form (the {{% %}} form ejects the nested content out of the list). +- name: index-not-optimized + content: | + Materialize indexes are not optimized for: + + - Ordered access, including: + + - Range queries, that is, queries using `>`, `>=`, `<`, `<=`, or `BETWEEN` + (e.g., `WHERE quantity > 10`, `WHERE price >= 10 AND price <= 50`, and + `WHERE quantity BETWEEN 10 AND 20`). + + - Queries that use `ORDER BY` on the index key. + + - Lookups on a prefix of a multi-column index key. For example, an index + with the key `(a, b)` is not optimized for a query that specifies an + equality condition on `a` but not on `b`. + + - Lookups that do not match the exact index key expression. For example, + for an index with the key `lower(a)`, an equality condition on `a` does + not match the index key; the query must specify an equality condition on + `lower(a)` for a point lookup. + + - `GROUP BY` aggregations. {{< include-from-yaml data="index_details" name="index-groupby-not-optimized" >}} diff --git a/doc/user/data/self_managed/self_managed_operator_compatibility.yml b/doc/user/data/self_managed/self_managed_operator_compatibility.yml index 96fda410754a0..fa91e90709909 100644 --- a/doc/user/data/self_managed/self_managed_operator_compatibility.yml +++ b/doc/user/data/self_managed/self_managed_operator_compatibility.yml @@ -5,6 +5,11 @@ columns: - column: Release date - column: Notes rows: + - Materialize Operator: v26.40 + orchestratord version: v26.40 + environmentd version: v26.40 + Release date: "2026-09-03" + Notes: "See [v26.40 release notes](/releases/#v26400)" - Materialize Operator: v26.39 orchestratord version: v26.39 environmentd version: v26.39 diff --git a/doc/user/static/images/indexes/index_in_memory.svg b/doc/user/static/images/indexes/index_in_memory.svg index ed6676610cc73..0def5002bd683 100644 --- a/doc/user/static/images/indexes/index_in_memory.svg +++ b/doc/user/static/images/indexes/index_in_memory.svg @@ -1,121 +1,114 @@ - - -In Memory - -cart_items_subtotals_idx -cart_totals_idx - - -hash -cart_id -user_id -cart_item_id -product_id -product_name -quantity -price -line_total -alert -7b4e1d52 -2 -3 -3 -7 -Rye Bread -1 -7.49 -7.49 -Notice: 0 in stock. -a3f2c891 -1 -2 -1 -3 -Baguette -1 -4.5 -4.5 -null -a3f2c891 -1 -2 -2 -6 -Chocolate Eclair -5 -4.99 -24.95 -null -f1c08e3a -3 -5 -6 -2 -Croissant -3 -3.49 -10.47 -null -f1c08e3a -3 -5 -4 -4 -Cinnamon Roll -3 -5.99 -17.97 -null -f1c08e3a -3 -5 -5 -5 -Blueberry Muffin -10 -3.99 -39.9 -null + + +In Memory + +cart_items_subtotals_idx +cart_totals_idx + + + + +cart_id +user_id +cart_item_id +product_id +product_name +quantity +price +line_total +alert +2 +3 +3 +7 +Rye Bread +1 +7.49 +7.49 +Notice: 0 in stock. +1 +2 +1 +3 +Baguette +1 +4.5 +4.5 +null +1 +2 +2 +6 +Chocolate Eclair +5 +4.99 +24.95 +null +3 +5 +6 +2 +Croissant +3 +3.49 +10.47 +null +3 +5 +4 +4 +Cinnamon Roll +3 +5.99 +17.97 +null +3 +5 +5 +5 +Blueberry Muffin +10 +3.99 +39.9 +null - - - - - - - - - - - - - - - - - - - - -hash -cart_id -cart_total -7b4e1d52 -2 -7.49 -a3f2c891 -1 -29.45 -f1c08e3a -3 -68.34 - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + +cart_id +cart_total +2 +7.49 +1 +29.45 +3 +68.34 + + + + + + + + + +index key column diff --git a/misc/python/materialize/parallel_workload/action.py b/misc/python/materialize/parallel_workload/action.py index 194d7b2f31516..fcb1f2a980898 100644 --- a/misc/python/materialize/parallel_workload/action.py +++ b/misc/python/materialize/parallel_workload/action.py @@ -1753,18 +1753,17 @@ def run(self, exe: Executor) -> bool: if index not in exe.db.indexes: return False - query = f"DROP INDEX {index}" - try: - exe.execute(query, http=Http.RANDOM) - except QueryError: - # The indexed object or its schema may have been dropped - # concurrently, taking the index with it. Untrack the index - # either way so stale entries don't fill up the set and choke - # off CreateIndexAction. Use discard, not remove: a concurrent - # CASCADE drop's untrack_objects_in_schemas may have already - # removed it, and remove would raise KeyError. - exe.db.indexes.discard(index) - raise + # The indexed object or its schema may have dropped + # concurrently, taking the index with it. Thus whether + # we drop it successfully or find it doesn't exist, we + # untrack its entry as to not fill up the set and choke + # off CreateIndexAction. An RBAC error however (in case + # a non-owner tries this drop) will raise an error and + # we'll retry the DropIndexAction. + exe.execute(f"DROP INDEX IF EXISTS {index}", http=Http.RANDOM) + # Use discard, not remove: a concurrent + # CASCADE drop's untrack_objects_in_schemas may have already + # removed it, and remove would raise KeyError. exe.db.indexes.discard(index) return True diff --git a/src/adapter/src/catalog.rs b/src/adapter/src/catalog.rs index b29d44a2ad12f..38d125ae21c56 100644 --- a/src/adapter/src/catalog.rs +++ b/src/adapter/src/catalog.rs @@ -2836,6 +2836,77 @@ mod tests { .await; } + /// Resolving a statement and resolving its normalized `create_sql` must + /// yield the same `ResolvedIds`. + /// + /// The in-memory catalog records the ids from the first resolution, while + /// a catalog reload re-derives them from the stored `create_sql`. Any + /// disagreement makes the reloaded catalog differ from the in-memory one. + #[mz_ore::test(tokio::test)] + #[cfg_attr(miri, ignore)] // slow + async fn test_resolved_ids_survive_create_sql_round_trip() { + use mz_ore::collections::CollectionExt; + Catalog::with_debug(|catalog| async move { + let conn_catalog = catalog.for_system_session(); + let scx = &mut StatementContext::new(None, &conn_catalog); + + let resolve_and_normalize = |scx: &mut StatementContext, sql: &str| { + let parsed = mz_sql_parser::parser::parse_statements(sql) + .expect("parses") + .into_element() + .ast; + let (stmt, ids) = names::resolve(scx.catalog, parsed).expect("resolves"); + let normalized = + mz_sql::normalize::create_statement(scx, stmt).expect("normalizes"); + (ids, normalized) + }; + + const ARRAY_SUFFIX: &str = "create table public.t (a pg_catalog.int4[])"; + const ARRAY_TYPE: &str = "create table public.t (a pg_catalog._int4)"; + const ELEMENT_TYPE: &str = "create table public.t (a pg_catalog.int4)"; + + let mut ids_by_spelling: BTreeMap<&str, Vec> = BTreeMap::new(); + for sql in [ + ARRAY_SUFFIX, + ARRAY_TYPE, + ELEMENT_TYPE, + "create table public.t (a pg_catalog.int4 list)", + "create view public.v as select null::pg_catalog.text[]", + ] { + let (ids, normalized) = resolve_and_normalize(scx, sql); + let (round_tripped_ids, _) = resolve_and_normalize(scx, &normalized); + assert_eq!( + ids.items().collect::>(), + round_tripped_ids.items().collect::>(), + "resolving {normalized:?} produced different ids than {sql:?}", + ); + ids_by_spelling.insert(sql, ids.items().copied().collect()); + } + + // `int4[]` and `_int4` name the same type, so both spellings must + // record the same ids. `int4[]` is only ever stored as `_int4`, + // so this is what keeps the reloaded catalog identical to the + // in-memory one. + assert_eq!( + ids_by_spelling[ARRAY_SUFFIX], ids_by_spelling[ARRAY_TYPE], + "{ARRAY_SUFFIX:?} and {ARRAY_TYPE:?} must resolve to the same ids", + ); + + // Neither spelling records the element type: an array reference + // names the array type alone. + let element_ids = &ids_by_spelling[ELEMENT_TYPE]; + assert!( + ids_by_spelling[ARRAY_SUFFIX] + .iter() + .all(|id| !element_ids.contains(id)), + "{ARRAY_SUFFIX:?} recorded an element type id from {ELEMENT_TYPE:?}", + ); + + catalog.expire().await; + }) + .await; + } + // Test that if a large catalog item is somehow committed, then we can still load the catalog. #[mz_ore::test(tokio::test)] #[cfg_attr(miri, ignore)] // slow diff --git a/src/adapter/src/catalog/builtin_table_updates.rs b/src/adapter/src/catalog/builtin_table_updates.rs index b71258e3898f0..b05dc8a6a1de4 100644 --- a/src/adapter/src/catalog/builtin_table_updates.rs +++ b/src/adapter/src/catalog/builtin_table_updates.rs @@ -18,10 +18,10 @@ use mz_catalog::builtin::{ BuiltinTable, MZ_AGGREGATES, MZ_ARRAY_TYPES, MZ_BASE_TYPES, MZ_CLUSTER_REPLICA_SIZE_INTERNAL, MZ_CLUSTER_REPLICA_SIZES, MZ_COLUMNS, MZ_EGRESS_IPS, MZ_FUNCTIONS, MZ_HISTORY_RETENTION_STRATEGIES, MZ_INDEX_COLUMNS, MZ_LICENSE_KEYS, MZ_LIST_TYPES, - MZ_MAP_TYPES, MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_OBJECT_DEPENDENCIES, - MZ_OBJECT_GLOBAL_IDS, MZ_OPERATORS, MZ_PSEUDO_TYPES, MZ_REPLACEMENTS, MZ_ROLE_AUTH, - MZ_SESSIONS, MZ_SOURCE_REFERENCES, MZ_STORAGE_USAGE_BY_SHARD, MZ_SUBSCRIPTIONS, - MZ_TYPE_PG_METADATA, MZ_TYPES, MZ_WEBHOOKS_SOURCES, + MZ_MAP_TYPES, MZ_MATERIALIZED_VIEW_REFRESH_STRATEGIES, MZ_OBJECT_GLOBAL_IDS, MZ_OPERATORS, + MZ_PSEUDO_TYPES, MZ_REPLACEMENTS, MZ_ROLE_AUTH, MZ_SESSIONS, MZ_SOURCE_REFERENCES, + MZ_STORAGE_USAGE_BY_SHARD, MZ_SUBSCRIPTIONS, MZ_TYPE_PG_METADATA, MZ_TYPES, + MZ_WEBHOOKS_SOURCES, }; use mz_catalog::durable::SourceReferences; use mz_catalog::memory::error::Error; @@ -103,19 +103,6 @@ impl CatalogState { BuiltinTableUpdate { id, data } } - pub fn pack_depends_update( - &self, - depender: CatalogItemId, - dependee: CatalogItemId, - diff: Diff, - ) -> BuiltinTableUpdate<&'static BuiltinTable> { - let row = Row::pack_slice(&[ - Datum::String(&depender.to_string()), - Datum::String(&dependee.to_string()), - ]); - BuiltinTableUpdate::row(&*MZ_OBJECT_DEPENDENCIES, row, diff) - } - pub(super) fn pack_role_auth_update( &self, id: RoleId, @@ -206,14 +193,6 @@ impl CatalogState { CatalogItem::Connection(_) => vec![], }; - if !entry.item().is_temporary() { - // Populate or clean up the `mz_object_dependencies` table. - // TODO(jkosh44) Unclear if this table wants to include all uses or only references. - for dependee in entry.item().references().items() { - updates.push(self.pack_depends_update(id, *dependee, diff)) - } - } - // Always report the latest for an objects columns. if let Some(desc) = entry.relation_desc_latest() { let defaults = match entry.item() { diff --git a/src/adapter/src/catalog/open/builtin_schema_migration.rs b/src/adapter/src/catalog/open/builtin_schema_migration.rs index 1754ccbf26250..34653cd13aad0 100644 --- a/src/adapter/src/catalog/open/builtin_schema_migration.rs +++ b/src/adapter/src/catalog/open/builtin_schema_migration.rs @@ -451,6 +451,17 @@ static MIGRATIONS: LazyLock> = LazyLock::new(|| { MZ_CATALOG_SCHEMA, "mz_sources", ), + // Converting mz_object_dependencies from a builtin table to a + // materialized view over mz_catalog_raw changes its catalog + // fingerprint, so it needs an explicit replacement step. See the + // NOTE above: this version must stay at the workspace's current dev + // version until the change ships. + MigrationStep::replacement( + "26.41.0-dev.0", + CatalogItemType::MaterializedView, + MZ_INTERNAL_SCHEMA, + "mz_object_dependencies", + ), ] }); diff --git a/src/catalog/src/builtin.rs b/src/catalog/src/builtin.rs index e70d02816250f..fae6d3e356974 100644 --- a/src/catalog/src/builtin.rs +++ b/src/catalog/src/builtin.rs @@ -31,6 +31,8 @@ mod mz_catalog; pub use mz_catalog::*; mod mz_internal; pub use mz_internal::*; +mod mz_object_dependencies; +pub use mz_object_dependencies::*; mod mz_introspection; pub use mz_introspection::*; mod information_schema; @@ -221,25 +223,20 @@ fn is_false(v: &bool) -> bool { /// /// Choosing the right variant matters: /// -/// - [`LinkProperties::ForeignKey`]: the source entity has a column whose -/// value is an ID that directly references a row in the target entity. -/// Use this when there is an explicit FK column (e.g. `schema_id` -> -/// `schema`). -/// - [`LinkProperties::DependsOn`]: this entity logically depends on the -/// target entity via a graph-edge table (e.g. `mz_compute_dependencies` -/// records that a compute object depends on another object). The -/// `source_column` is the column **in this entity** that holds the -/// dependent's ID; `target_column` is the column in the target entity -/// being depended upon. Use this for dependency-graph tables, **not** -/// `ForeignKey`. -/// - [`LinkProperties::Union`]: the source entity is a superset view that -/// contains the target entity as a subset, optionally filtered by a -/// discriminator column. -/// - [`LinkProperties::MapsTo`]: the source entity provides an ID translation -/// to the target entity, possibly via an intermediate table or across ID -/// namespaces. -/// - [`LinkProperties::Measures`]: the source entity records metric -/// measurements about the target entity. +/// - [`LinkProperties::ForeignKey`]: the source entity has a column whose value is an ID that +/// directly references a row in the target entity. Use this when there is an explicit FK column +/// (e.g. `schema_id` -> `schema`). +/// - [`LinkProperties::DependsOn`]: this entity logically depends on the target entity via a +/// graph-edge table (e.g. `mz_compute_dependencies` records that a compute object depends on +/// another object). The `source_column` is the column **in this entity** that holds the +/// dependent's ID; `target_column` is the column in the target entity being depended upon. Use +/// this for dependency-graph tables, **not** `ForeignKey`. +/// - [`LinkProperties::Union`]: the source entity is a superset view that contains the target +/// entity as a subset, optionally filtered by a discriminator column. +/// - [`LinkProperties::MapsTo`]: the source entity provides an ID translation to the target entity, +/// possibly via an intermediate table or across ID namespaces. +/// - [`LinkProperties::Measures`]: the source entity records metric measurements about the target +/// entity. #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, serde::Serialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum LinkProperties { @@ -783,6 +780,20 @@ impl Fingerprint for SqlRelationType { } } +/// Asserts that `name` is safe to embed unquoted inside a `'...'`-quoted SQL literal +/// or inside a `"..."`-quoted SQL identifier. Generated builtin relations +/// (`make_mz_indexes`, `make_mz_object_dependencies_raw`, ...) concatenate builtin names +/// into SQL fragments, so a quote or backslash would produce malformed SQL. Builtin +/// names should always be plain ASCII identifiers. +pub(super) fn assert_safe_builtin_name(name: &str, kind: &str) { + assert!( + !name.contains('\'') && !name.contains('"') && !name.contains('\\'), + "builtin {kind} name {name:?} contains an unsupported character; \ + generated builtin relations reconstruct SQL via string \ + concatenation and assume names contain no quotes or backslashes" + ); +} + pub(super) const PUBLIC_SELECT: MzAclItem = MzAclItem { grantee: RoleId::Public, grantor: MZ_SYSTEM_ROLE_ID, @@ -1137,7 +1148,9 @@ pub static BUILTINS_STATIC: LazyLock>> = LazyLock::ne Builtin::MaterializedView(&MZ_KAFKA_SINKS), Builtin::MaterializedView(&MZ_KAFKA_CONNECTIONS), Builtin::MaterializedView(&MZ_KAFKA_SOURCES), - Builtin::Table(&MZ_OBJECT_DEPENDENCIES), + // mz_object_dependencies_raw is generated dynamically below with inlined + // builtin VALUES and inserted directly before this entry. + Builtin::MaterializedView(&MZ_OBJECT_DEPENDENCIES), Builtin::MaterializedView(&MZ_ICEBERG_SINKS), Builtin::MaterializedView(&MZ_DATABASES), Builtin::MaterializedView(&MZ_SCHEMAS), @@ -1555,6 +1568,38 @@ pub static BUILTINS_STATIC: LazyLock>> = LazyLock::ne builtin_items.insert(insert_pos, Builtin::MaterializedView(mz_indexes_ref)); } + // Generate mz_object_dependencies_raw, which inlines every builtin's + // dependency edges as VALUES. + // + // It has to run before ontology::generate_views and builtin::builtins, because those consume + // mz_object_dependencies. However, because the views those two generators produce + // have dependency edges of their own, it must use them as inputs. So this block runs both + // generators early, keeps the edges of the views they hand back, and discards the views + // themselves. The real ones are built below. + { + let ontology_preview = ontology::generate_views(&builtin_items); + let builtin_reporter_preview: Vec<_> = builtin::builtins(&builtin_items).collect(); + let generator_input: Vec<_> = builtin_types + .iter() + .chain(builtin_funcs.iter()) + .chain(builtin_reporter_preview.iter()) + .chain(builtin_items.iter()) + .chain(ontology_preview.iter()) + .cloned() + .collect(); + let mz_object_dependencies_raw = + mz_object_dependencies::make_mz_object_dependencies_raw(&generator_input); + let mz_object_dependencies_raw_ref: &'static BuiltinView = + Box::leak(Box::new(mz_object_dependencies_raw)); + // The view goes directly before mz_object_dependencies, which reads it, + // because BUILTINS_STATIC must list dependencies before their dependents. + let insert_pos = builtin_items + .iter() + .position(|b| b.name() == "mz_object_dependencies") + .expect("mz_object_dependencies must be present in builtin_items"); + builtin_items.insert(insert_pos, Builtin::View(mz_object_dependencies_raw_ref)); + } + // Generate ontology views by enumerating existing builtins. builtin_items.extend(ontology::generate_views(&builtin_items)); @@ -2004,8 +2049,8 @@ mod tests { // avoid noise. // // Exemptions: - // - Column at index 0 named "id": almost always the entity's own PK, - // not a FK (e.g. mz_objects.id, mz_functions.id). + // - Column at index 0 named "id": almost always the entity's own PK, not a FK (e.g. + // mz_objects.id, mz_functions.id). // - Columns in the relation's declared key set. // // "Reference" types are ID types that imply a FK. Discriminators @@ -2429,4 +2474,40 @@ mod tests { "mz_indexes fingerprint must change when a builtin log is added" ); } + + /// Because `mz_object_dependencies_raw`` is built from prior copies of the ontology and builtin + /// reporter views to get their edges since they must be built afterwards. Thus we test + /// that regenerating the view using all builtins (excluding itself) equals the SQL of the real + /// view. + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // slow: parses every builtin's SQL twice + fn test_mz_object_dependencies_raw_sql_is_stable() { + let without_self: Vec> = BUILTINS_STATIC + .iter() + .filter(|b| { + !matches!(b, Builtin::View(v) if v.name == mz_object_dependencies::MZ_OBJECT_DEPENDENCIES_RAW) + }) + .cloned() + .collect(); + + let regenerated = mz_object_dependencies::make_mz_object_dependencies_raw(&without_self); + + let from_static = BUILTINS_STATIC + .iter() + .find_map(|b| match b { + Builtin::View(v) + if v.name == mz_object_dependencies::MZ_OBJECT_DEPENDENCIES_RAW => + { + Some(*v) + } + _ => None, + }) + .expect("mz_object_dependencies_raw must be present in BUILTINS_STATIC"); + + assert_eq!( + regenerated.sql, from_static.sql, + "regenerating mz_object_dependencies_raw from the final builtin \ + list must reproduce the SQL generated during static init" + ); + } } diff --git a/src/catalog/src/builtin/mz_catalog.rs b/src/catalog/src/builtin/mz_catalog.rs index 3874ac8d7d177..bdd9cd3969b5e 100644 --- a/src/catalog/src/builtin/mz_catalog.rs +++ b/src/catalog/src/builtin/mz_catalog.rs @@ -29,6 +29,7 @@ use mz_storage_client::controller::IntrospectionType; use super::{ BuiltinIndex, BuiltinLog, BuiltinMaterializedView, BuiltinSource, BuiltinTable, BuiltinType, BuiltinView, Cardinality, LinkProperties, Ontology, OntologyLink, PUBLIC_SELECT, + assert_safe_builtin_name, }; pub const TYPE_LIST: BuiltinType = BuiltinType { @@ -676,19 +677,6 @@ pub static MZ_COLUMNS: LazyLock = LazyLock::new(|| BuiltinTable { }); // mz_indexes is generated dynamically in BUILTINS_STATIC via mz_catalog::make_mz_indexes() -/// Asserts that `name` is safe to embed unquoted inside a `'...'`-quoted SQL literal -/// or inside a `"..."`-quoted SQL identifier. Builtin index/log/object names are -/// concatenated into SQL fragments below, so a quote or backslash would produce -/// malformed SQL. Builtin names should always be plain ASCII identifiers. -fn assert_safe_builtin_name(name: &str, kind: &str) { - assert!( - !name.contains('\'') && !name.contains('"') && !name.contains('\\'), - "builtin {kind} name {name:?} contains an unsupported character; \ - mz_indexes reconstructs SQL via string concatenation and assumes \ - names contain no quotes or backslashes" - ); -} - /// User-created indexes, sourced from `mz_catalog_raw` `Item` entries with /// `parse_catalog_create_sql(...)` yielding `type = 'index'`. const USER_INDEXES_CTE: &str = "\ diff --git a/src/catalog/src/builtin/mz_internal.rs b/src/catalog/src/builtin/mz_internal.rs index e8e154850df90..006492fbdf5d2 100644 --- a/src/catalog/src/builtin/mz_internal.rs +++ b/src/catalog/src/builtin/mz_internal.rs @@ -394,64 +394,7 @@ WHERE }), } }); -pub static MZ_OBJECT_DEPENDENCIES: LazyLock = LazyLock::new(|| BuiltinTable { - name: "mz_object_dependencies", - schema: MZ_INTERNAL_SCHEMA, - oid: oid::TABLE_MZ_OBJECT_DEPENDENCIES_OID, - desc: RelationDesc::builder() - .with_column("object_id", SqlScalarType::String.nullable(false)) - .with_column( - "referenced_object_id", - SqlScalarType::String.nullable(false), - ) - .finish(), - column_comments: BTreeMap::from_iter([ - ( - "object_id", - "The ID of the dependent object. Corresponds to `mz_objects.id`.", - ), - ( - "referenced_object_id", - "The ID of the referenced object. Corresponds to `mz_objects.id`.", - ), - ]), - is_retained_metrics_object: true, - access: vec![PUBLIC_SELECT], - ontology: Some(Ontology { - entity_name: "object_dependency", - description: "A dependency edge: one object depends on another", - links: &const { - [ - OntologyLink { - name: "depends_on", - target: "object", - properties: LinkProperties::DependsOn { - source_column: "object_id", - target_column: "id", - source_id_type: Some(mz_repr::SemanticType::CatalogItemId), - requires_mapping: None, - }, - }, - OntologyLink { - name: "dependency_is", - target: "object", - properties: LinkProperties::DependsOn { - source_column: "referenced_object_id", - target_column: "id", - source_id_type: Some(mz_repr::SemanticType::CatalogItemId), - requires_mapping: None, - }, - }, - ] - }, - column_semantic_types: &const { - [ - ("object_id", SemanticType::CatalogItemId), - ("referenced_object_id", SemanticType::CatalogItemId), - ] - }, - }), -}); + pub static MZ_COMPUTE_DEPENDENCIES: LazyLock = LazyLock::new(|| BuiltinSource { name: "mz_compute_dependencies", schema: MZ_INTERNAL_SCHEMA, @@ -4979,9 +4922,10 @@ ON mz_internal.pg_attrdef_all_databases (oid, adrelid, adnum, adbin, adsrc)", pub static MZ_COMPUTE_ERROR_COUNTS_RAW_UNIFIED: LazyLock = LazyLock::new(|| BuiltinSource { - // TODO(database-issues#8173): Rename this source to `mz_compute_error_counts_raw`. Currently this causes a - // naming conflict because the resolver stumbles over the source with the same name in - // `mz_introspection` due to the automatic schema translation. + // TODO(database-issues#8173): Rename this source to `mz_compute_error_counts_raw`. + // Currently this causes a naming conflict because the resolver stumbles over the + // source with the same name in `mz_introspection` due to the automatic schema + // translation. name: "mz_compute_error_counts_raw_unified", schema: MZ_INTERNAL_SCHEMA, oid: oid::SOURCE_MZ_COMPUTE_ERROR_COUNTS_RAW_UNIFIED_OID, @@ -8000,12 +7944,11 @@ fn console_cluster_utilization_overview_desc() -> RelationDesc { /// `console/src/api/materialize/cluster/replicaUtilizationHistory.ts`). /// /// * `bin`: the `date_bin` bucket width, e.g. `1 MINUTE`. -/// * `retention`: how much history the view retains, e.g. `3 HOURS`, enforced -/// with a temporal `mz_now()` filter so the maintained arrangement stays -/// bounded. -/// * `group_size`: the expected number of metric samples per (replica, bucket), -/// used for the `DISTINCT ON INPUT GROUP SIZE` top-k hint. Replica metrics are -/// scraped roughly once per minute, so this is the bucket width in minutes. +/// * `retention`: how much history the view retains, e.g. `3 HOURS`, enforced with a temporal +/// `mz_now()` filter so the maintained arrangement stays bounded. +/// * `group_size`: the expected number of metric samples per (replica, bucket), used for the +/// `DISTINCT ON INPUT GROUP SIZE` top-k hint. Replica metrics are scraped roughly once per +/// minute, so this is the bucket width in minutes. fn console_cluster_utilization_overview_sql(bin: &str, retention: &str, group_size: u32) -> String { format!( r#"WITH replica_history AS ( @@ -8359,12 +8302,11 @@ pub static MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H: LazyLock = * cluster_name: The name of the cluster. * The approach taken is as follows. First, find all extant clusters and add them * to the result set. Per cluster, we do the following: - * 1. Find the most recent create or rename event. This moment represents when the - * cluster took on its final logical identity. - * 2. Look for a cluster that had the same name (or the same name with `_dbt_deploy` - * appended) that was dropped within one minute of that moment. That cluster is - * almost certainly the logical predecessor of the current cluster. Add the cluster - * to the result set. + * 1. Find the most recent create or rename event. This moment represents when the cluster took + * on its final logical identity. + * 2. Look for a cluster that had the same name (or the same name with `_dbt_deploy` appended) + * that was dropped within one minute of that moment. That cluster is almost certainly the + * logical predecessor of the current cluster. Add the cluster to the result set. * 3. Repeat the procedure until a cluster with no logical predecessor is discovered. * Limiting the search for a dropped cluster to a window of one minute is a heuristic, * but one that's likely to be pretty good one. If a name is reused after more @@ -8728,8 +8670,9 @@ ON mz_internal.mz_sink_status_history (sink_id)", // underlying relation. // // We append WITH_HISTORY because we want to build a separate view + index that doesn't -// retain history. This is because retaining its history causes MZ_SOURCE_STATISTICS_WITH_HISTORY_IND -// to hold all records/updates, which causes CPU and latency of querying it to spike. +// retain history. This is because retaining its history causes +// MZ_SOURCE_STATISTICS_WITH_HISTORY_IND to hold all records/updates, which causes CPU and latency +// of querying it to spike. pub static MZ_SOURCE_STATISTICS_WITH_HISTORY: LazyLock = LazyLock::new(|| BuiltinView { name: "mz_source_statistics_with_history", diff --git a/src/catalog/src/builtin/mz_object_dependencies.rs b/src/catalog/src/builtin/mz_object_dependencies.rs new file mode 100644 index 0000000000000..a03d46a87dc56 --- /dev/null +++ b/src/catalog/src/builtin/mz_object_dependencies.rs @@ -0,0 +1,465 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +//! The `mz_internal.mz_object_dependencies` materialized view and the +//! generated view it reads. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::LazyLock; + +use itertools::Itertools; +use mz_catalog_protos::objects::CatalogItemType as ProtoCatalogItemType; +use mz_ore::collections::CollectionExt; +use mz_pgrepr::oid; +use mz_repr::namespaces::MZ_INTERNAL_SCHEMA; +use mz_repr::{RelationDesc, SemanticType, SqlScalarType}; +use mz_sql::catalog::{CatalogType, NameReference}; +use mz_sql_parser::ast::UnresolvedItemName; +use mz_sql_parser::ast::item_refs::collect_item_references; + +use super::{ + Builtin, BuiltinMaterializedView, BuiltinView, LinkProperties, Ontology, OntologyLink, + PUBLIC_SELECT, assert_safe_builtin_name, +}; + +pub(super) const MZ_OBJECT_DEPENDENCIES_RAW: &str = "mz_object_dependencies_raw"; + +/// The durable encoding of a `GidMapping` key's `object_type`, as it appears in +/// `mz_catalog_raw`'s JSON. +fn object_type_code(object_type: ProtoCatalogItemType) -> String { + serde_json::to_string(&object_type).expect("CatalogItemType is serializable") +} + +/// One builtin dependency edge, inlined as a `VALUES` row of +/// `mz_object_dependencies_raw`. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct BuiltinEdgeRow { + object_schema: String, + object_name: String, + object_type: String, + ref_schema: String, + ref_name: String, + ref_kind: &'static str, +} + +impl BuiltinEdgeRow { + /// Renders the row as a SQL `VALUES` tuple. + /// + /// The literals are not escaped. `BuiltinEdgeCollector::collect` runs + /// every name through `assert_safe_builtin_name` before building a row. + fn to_sql_row(&self) -> String { + let BuiltinEdgeRow { + object_schema, + object_name, + object_type, + ref_schema, + ref_name, + ref_kind, + } = self; + format!( + "('{object_schema}', '{object_name}', '{object_type}', \ + '{ref_schema}', '{ref_name}', '{ref_kind}')" + ) + } +} + +/// Collects builtin dependency edges by parsing builtin SQL. +struct BuiltinEdgeCollector<'a> { + /// Builtin name to its schema, for resolving unqualified references. + schema_by_name: BTreeMap<&'a str, &'a str>, + /// Element type name to the `(schema, name)` of its array type. e.g. `T[]` + /// maps to the qualified Postgres compatible type `_T`. + array_type_by_elem: BTreeMap<&'a str, (&'a str, &'a str)>, + rows: BTreeSet, +} + +impl<'a> BuiltinEdgeCollector<'a> { + fn new(builtins: &'a [Builtin]) -> Self { + let mut schema_by_name: BTreeMap<&str, &str> = BTreeMap::new(); + for b in builtins { + if let Some(prev) = schema_by_name.insert(b.name(), b.schema()) { + assert_eq!( + prev, + b.schema(), + "builtin name {} appears in multiple schemas; unqualified references are ambiguous", + b.name() + ); + } + } + + let mut array_type_by_elem: BTreeMap<&str, (&str, &str)> = BTreeMap::new(); + for b in builtins { + if let Builtin::Type(t) = b { + if let CatalogType::Array { element_reference } = &t.details.typ { + array_type_by_elem.insert(element_reference, (t.schema, t.name)); + } + } + } + + BuiltinEdgeCollector { + schema_by_name, + array_type_by_elem, + rows: BTreeSet::new(), + } + } + + fn resolve(&self, object: &str, name: &UnresolvedItemName) -> (String, String) { + match &name.0[..] { + [.., schema, item] => (schema.as_str().to_string(), item.as_str().to_string()), + [item] => { + let item = item.as_str(); + let schema = self.schema_by_name.get(item).unwrap_or_else(|| { + panic!( + "cannot resolve unqualified reference {item:?} in builtin {object}; \ + qualify the reference or check that the referenced builtin exists" + ) + }); + (schema.to_string(), item.to_string()) + } + [] => panic!("empty item reference in builtin {object}"), + } + } + + /// Parses `create_sql` and records one row per catalog item it references. + fn collect( + &mut self, + object_schema: &str, + object_name: &str, + object_type: ProtoCatalogItemType, + create_sql: &str, + ) { + assert_safe_builtin_name(object_schema, "object schema"); + assert_safe_builtin_name(object_name, "object"); + + let stmt = mz_sql::parse::parse(create_sql) + .unwrap_or_else(|e| panic!("invalid sql for builtin {object_name}: {e}")) + .into_element() + .ast; + let refs = collect_item_references(&stmt); + + // Builtin SQL is name-based. An id reference here means somebody + // hardcoded a system id, which is not stable across versions. + assert!( + refs.ids.is_empty(), + "builtin {} references items by id: {:?}", + object_name, + refs.ids + ); + + let mut referenced: Vec<(String, String, &'static str)> = Vec::new(); + for item_name in &refs.named_relations { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "rel")); + } + for item_name in &refs.named_funcs { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "func")); + } + for item_name in &refs.named_types { + let (schema, name) = self.resolve(object_name, item_name); + referenced.push((schema, name, "type")); + } + // Map the array type `T[]` to its Postgres compatible type `_T`. + for elem in &refs.named_array_elements { + // `T[]` references the array type paired with `T`, not `T` itself. + let (_, name) = self.resolve(object_name, elem); + let (array_schema, array_name) = self + .array_type_by_elem + .get(name.as_str()) + .unwrap_or_else(|| { + panic!( + "builtin {object_name} uses {name}[], but no builtin array type has \ + element {name}" + ) + }); + referenced.push((array_schema.to_string(), array_name.to_string(), "type")); + } + + let object_type = object_type_code(object_type); + for (ref_schema, ref_name, ref_kind) in referenced { + assert_safe_builtin_name(&ref_schema, "referenced schema"); + assert_safe_builtin_name(&ref_name, "referenced object"); + assert!( + !(object_schema == ref_schema && object_name == ref_name), + "builtin {object_schema}.{object_name} references itself" + ); + self.rows.insert(BuiltinEdgeRow { + object_schema: object_schema.to_string(), + object_name: object_name.to_string(), + object_type: object_type.clone(), + ref_schema, + ref_name, + ref_kind, + }); + } + } +} + +/// Renders the `mz_object_dependencies_raw` view body with `rows` inlined as +/// the builtin edge table. +fn mz_object_dependencies_raw_sql(rows: &BTreeSet) -> String { + let builtin_ref_values = rows.iter().map(BuiltinEdgeRow::to_sql_row).join(","); + + let func_type = object_type_code(ProtoCatalogItemType::Func); + let type_type = object_type_code(ProtoCatalogItemType::Type); + let source_type = object_type_code(ProtoCatalogItemType::Source); + // Every kind a name in relation position can denote. + let relation_types = format!("NOT IN ('{type_type}', '{func_type}')"); + + format!( + " +WITH + user_items AS ( + SELECT + mz_internal.parse_catalog_id(data->'key'->'gid') AS id, + mz_internal.parse_catalog_item_references(data->'value'->'definition'->'V1'->>'create_sql') AS refs + FROM mz_internal.mz_catalog_raw + WHERE + data->>'kind' = 'Item' AND + -- Exclude temporary objects + data->'value'->>'ephemeral_owner_session' IS NULL + ), + gid_mappings AS ( + SELECT + 's' || (data->'value'->>'catalog_id') AS id, + data->'key'->>'schema_name' AS schema_name, + data->'key'->>'object_name' AS object_name, + data->'key'->>'object_type' AS object_type + FROM mz_internal.mz_catalog_raw + WHERE data->>'kind' = 'GidMapping' + ), + user_id_edges AS ( + SELECT u.id AS object_id, r.ref AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements_text(u.refs->'ids') AS r(ref) + ), + user_func_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_funcs') AS f(func) + JOIN gid_mappings gm ON + gm.object_type = '{func_type}' AND + gm.schema_name = f.func->>'schema' AND + gm.object_name = f.func->>'name' + ), + user_type_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_types') AS t(typ) + JOIN gid_mappings gm ON + gm.object_type = '{type_type}' AND + gm.schema_name = t.typ->>'schema' AND + gm.object_name = t.typ->>'name' + ), + user_relation_edges AS ( + SELECT u.id AS object_id, gm.id AS referenced_object_id + FROM user_items u + CROSS JOIN LATERAL jsonb_array_elements(u.refs->'named_relations') AS n(rel) + JOIN gid_mappings gm ON + gm.object_type {relation_types} AND + gm.schema_name = n.rel->>'schema' AND + gm.object_name = n.rel->>'name' + ), + builtin_edges AS ( + SELECT obj.id AS object_id, ref.id AS referenced_object_id + FROM + (VALUES {builtin_ref_values}) + AS bv(object_schema, object_name, object_type, ref_schema, ref_name, ref_kind) + JOIN gid_mappings obj ON + obj.schema_name = bv.object_schema AND + obj.object_name = bv.object_name AND + obj.object_type = bv.object_type + JOIN gid_mappings ref ON + ref.schema_name = bv.ref_schema AND + ref.object_name = bv.ref_name AND + CASE bv.ref_kind + WHEN 'func' THEN ref.object_type = '{func_type}' + WHEN 'type' THEN ref.object_type = '{type_type}' + ELSE ref.object_type {relation_types} + END + ), + introspection_source_index_edges AS ( + SELECT + 'si' || (isi.data->'value'->>'catalog_id') AS object_id, + 's' || (gm.data->'value'->>'catalog_id') AS referenced_object_id + FROM mz_internal.mz_catalog_raw AS isi + JOIN mz_internal.mz_catalog_raw AS gm ON + gm.data->>'kind' = 'GidMapping' AND + gm.data->'key'->>'object_type' = '{source_type}' AND + gm.data->'key'->>'schema_name' = 'mz_introspection' AND + gm.data->'key'->>'object_name' = isi.data->'key'->>'name' + WHERE isi.data->>'kind' = 'ClusterIntrospectionSourceIndex' + ) +-- UNION rather than UNION ALL: the branches are disjoint for the most part, but a +-- reference can appear both by id and by name in one statement (a table function +-- spelled `FROM [sNN AS pg_catalog.generate_series](...)` next to a plain call to +-- the same function, which resolves by name) +SELECT object_id, referenced_object_id FROM user_id_edges +UNION +SELECT object_id, referenced_object_id FROM user_func_edges +UNION +SELECT object_id, referenced_object_id FROM user_type_edges +UNION +SELECT object_id, referenced_object_id FROM user_relation_edges +UNION +SELECT object_id, referenced_object_id FROM builtin_edges +UNION +SELECT object_id, referenced_object_id FROM introspection_source_index_edges +" + ) +} + +/// Generate the `mz_internal.mz_object_dependencies_raw` builtin view with +/// builtin dependency edges inlined as VALUES clauses. +/// +/// The view unions these edge sources: +/// +/// - User items: references extracted from stored `create_sql` via `parse_catalog_item_references`. +/// Id references are used directly. Function references are recovered by joining +/// `(schema, name)` against `GidMapping` rows, which is sound by construction: name resolution +/// never prints an id for a function (`print_id` is false for `Func`) and a function's +/// `full_name` always carries its schema, so a function reference persists as exactly +/// `"schema"."name"`. Temporary items are excluded. +/// - Builtin items: name references joined to `GidMapping` at query time. +/// - Introspection source indexes: `si -> s` edges from +/// `ClusterIntrospectionSourceIndex` entries. +pub(super) fn make_mz_object_dependencies_raw(builtins: &[Builtin]) -> BuiltinView { + let mut collector = BuiltinEdgeCollector::new(builtins); + for b in builtins { + let (object_type, create_sql) = match b { + Builtin::View(v) => (ProtoCatalogItemType::View, v.create_sql()), + Builtin::MaterializedView(mv) => { + (ProtoCatalogItemType::MaterializedView, mv.create_sql()) + } + Builtin::Index(i) => (ProtoCatalogItemType::Index, i.create_sql()), + // The user_items branch already covers runtime alterable connections, thus we skip as + // to not double-count. + Builtin::Connection(c) if c.runtime_alterable => continue, + Builtin::Connection(c) => (ProtoCatalogItemType::Connection, c.sql.to_string()), + // Skip objects constructed without SQL. + Builtin::Log(_) + | Builtin::Table(_) + | Builtin::Type(_) + | Builtin::Func(_) + | Builtin::Source(_) => continue, + }; + collector.collect(b.schema(), b.name(), object_type, &create_sql); + } + + // Append `mz_object_dependencies_raw`'s own outgoing edges to its VALUES + // rows. + let self_edges = |sql: &str| { + let mut collector = BuiltinEdgeCollector::new(builtins); + collector.collect( + MZ_INTERNAL_SCHEMA, + MZ_OBJECT_DEPENDENCIES_RAW, + ProtoCatalogItemType::View, + &format!("CREATE VIEW {MZ_INTERNAL_SCHEMA}.{MZ_OBJECT_DEPENDENCIES_RAW} AS {sql}"), + ); + collector.rows + }; + + let self_rows = self_edges(&mz_object_dependencies_raw_sql(&collector.rows)); + let mut rows = collector.rows; + rows.extend(self_rows.iter().cloned()); + let sql = mz_object_dependencies_raw_sql(&rows); + + BuiltinView { + name: MZ_OBJECT_DEPENDENCIES_RAW, + schema: MZ_INTERNAL_SCHEMA, + oid: oid::VIEW_MZ_OBJECT_DEPENDENCIES_RAW_OID, + desc: RelationDesc::builder() + .with_column("object_id", SqlScalarType::String.nullable(true)) + .with_column("referenced_object_id", SqlScalarType::String.nullable(true)) + .with_key(vec![0, 1]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "object_id", + "The ID of the dependent object. Corresponds to `mz_objects.id`.", + ), + ( + "referenced_object_id", + "The ID of the referenced object. Corresponds to `mz_objects.id`.", + ), + ]), + sql: Box::leak(sql.into_boxed_str()), + access: vec![PUBLIC_SELECT], + ontology: None, + } +} + +pub static MZ_OBJECT_DEPENDENCIES: LazyLock = + LazyLock::new(|| BuiltinMaterializedView { + name: "mz_object_dependencies", + schema: MZ_INTERNAL_SCHEMA, + oid: oid::MV_MZ_OBJECT_DEPENDENCIES_OID, + desc: RelationDesc::builder() + .with_column("object_id", SqlScalarType::String.nullable(false)) + .with_column( + "referenced_object_id", + SqlScalarType::String.nullable(false), + ) + .with_key(vec![0, 1]) + .finish(), + column_comments: BTreeMap::from_iter([ + ( + "object_id", + "The ID of the dependent object. Corresponds to `mz_objects.id`.", + ), + ( + "referenced_object_id", + "The ID of the referenced object. Corresponds to `mz_objects.id`.", + ), + ]), + sql: " +IN CLUSTER mz_catalog_server +WITH ( + ASSERT NOT NULL object_id, + ASSERT NOT NULL referenced_object_id +) AS +SELECT object_id, referenced_object_id +FROM mz_internal.mz_object_dependencies_raw", + is_retained_metrics_object: true, + access: vec![PUBLIC_SELECT], + ontology: Some(Ontology { + entity_name: "object_dependency", + description: "A dependency edge: one object depends on another", + links: &const { + [ + OntologyLink { + name: "depends_on", + target: "object", + properties: LinkProperties::DependsOn { + source_column: "object_id", + target_column: "id", + source_id_type: Some(mz_repr::SemanticType::CatalogItemId), + requires_mapping: None, + }, + }, + OntologyLink { + name: "dependency_is", + target: "object", + properties: LinkProperties::DependsOn { + source_column: "referenced_object_id", + target_column: "id", + source_id_type: Some(mz_repr::SemanticType::CatalogItemId), + requires_mapping: None, + }, + }, + ] + }, + column_semantic_types: &const { + [ + ("object_id", SemanticType::CatalogItemId), + ("referenced_object_id", SemanticType::CatalogItemId), + ] + }, + }), + }); diff --git a/src/compute-types/tests/snapshots/lir_v1.json b/src/compute-types/tests/snapshots/lir_v1.json index 311d1fbdb95c4..e15f7e8b893c5 100644 --- a/src/compute-types/tests/snapshots/lir_v1.json +++ b/src/compute-types/tests/snapshots/lir_v1.json @@ -3067,6 +3067,7 @@ "ParseCatalogAuditLogDetails": "UNITSTRUCT", "ParseCatalogCreateSql": "UNITSTRUCT", "ParseCatalogId": "UNITSTRUCT", + "ParseCatalogItemReferences": "UNITSTRUCT", "ParseCatalogPrivileges": "UNITSTRUCT", "ParseConnectionDetails": "UNITSTRUCT", "ParseIdent": "UNITSTRUCT", @@ -5086,6 +5087,11 @@ "TYPENAME": "ParseCatalogId" } }, + "ParseCatalogItemReferences": { + "NEWTYPE": { + "TYPENAME": "ParseCatalogItemReferences" + } + }, "ParseCatalogPrivileges": { "NEWTYPE": { "TYPENAME": "ParseCatalogPrivileges" diff --git a/src/expr/src/scalar/func/impls/jsonb.rs b/src/expr/src/scalar/func/impls/jsonb.rs index 287027f3eedca..f21b7109b1b00 100644 --- a/src/expr/src/scalar/func/impls/jsonb.rs +++ b/src/expr/src/scalar/func/impls/jsonb.rs @@ -17,6 +17,7 @@ use mz_repr::adt::numeric::{self, Numeric, NumericMaxScale}; use mz_repr::role_id::RoleId; use mz_repr::{ArrayRustType, Datum, Row, RowPacker, SqlColumnType, SqlScalarType, strconv}; use mz_sql_parser::ast::display::AstDisplay; +use mz_sql_parser::ast::item_refs::collect_item_references; use mz_sql_parser::ast::{ AstInfo, AvroSchema, ConnectionOption, ConnectionOptionName, CreateConnectionType, CreateSinkConnection, CreateSubsourceOptionName, Format, FormatSpecifier, @@ -408,7 +409,6 @@ fn option_string(value: &WithOptionValue) -> Option { /// /// The returned JSONB does not fully reflect the parsed SQL and instead contains only fields /// required by current callers. -/// // TODO: This function isn't parsing JSONB and therefore shouldn't live in the `jsonb` module. // Consider moving all the `parse_catalog_*` functions into their own module. #[sqlfunc] @@ -815,6 +815,51 @@ fn parse_catalog_create_sql<'a>(a: &'a str) -> Result { Ok(jsonb) } +/// Extracts the catalog item references from a catalog `create_sql` string as a JSONB object. +/// +/// - `ids`: array of catalog id strings from `[ AS ]` references. +/// - `named_funcs`: array of function references such as "pg_catalog"."max". +/// - `named_types`: array of type references such as "pg_catalog"."int4", exclusive from the ones +/// in `ids`. +/// - `named_relations`: array of relation references, exclusive from the ones in `ids`. +#[sqlfunc] +fn parse_catalog_item_references<'a>(a: &'a str) -> Result { + fn qualified(name: &UnresolvedItemName) -> serde_json::Value { + match &name.0[..] { + [.., schema, item] => json!({"schema": schema.as_str(), "name": item.as_str()}), + [item] => json!({"schema": serde_json::Value::Null, "name": item.as_str()}), + [] => json!({"schema": serde_json::Value::Null, "name": ""}), + } + } + + let parse = || -> Result { + let mut stmts = mz_sql_parser::parser::parse_statements(a) + .map_err(|e| format!("failed to parse create_sql: {e}"))?; + let stmt = match stmts.len() { + 1 => stmts.remove(0).ast, + n => return Err(format!("expected a single statement, found {n}")), + }; + + let refs = collect_item_references(&stmt); + mz_ore::soft_assert_or_log!( + refs.named_array_elements.is_empty(), + "persisted create_sql should never carry an array type T[] \ + and resolve its array type in `ids`: {:?}", + refs.named_array_elements + ); + Ok(json!({ + "ids": refs.ids.iter().collect::>(), + "named_funcs": refs.named_funcs.iter().map(qualified).collect::>(), + "named_types": refs.named_types.iter().map(qualified).collect::>(), + "named_relations": refs.named_relations.iter().map(qualified).collect::>(), + })) + }; + + let val = parse().map_err(|e| EvalError::InvalidCatalogJson(e.into()))?; + let jsonb = Jsonb::from_serde_json(val).expect("valid JSONB"); + Ok(jsonb) +} + /// Minimal decoder for `ProtoPostgresSourcePublicationDetails`. The /// canonical proto lives in `mz-storage-types`, which depends on /// `mz-expr`, so we redeclare the two tags we read here. Upstream tag @@ -831,8 +876,8 @@ struct PostgresPublicationDetailsSubset { /// Extracts postgres source publication details (slot, timeline_id) from a /// catalog `create_sql`. Returns: /// -/// - jsonb `{"slot": , "timeline_id": }` for -/// `CREATE SOURCE ... FROM POSTGRES CONNECTION ... (DETAILS = ...)` statements. +/// - jsonb `{"slot": , "timeline_id": }` for `CREATE SOURCE ... FROM POSTGRES +/// CONNECTION ... (DETAILS = ...)` statements. /// - jsonb `null` for any other statement. /// /// Errors if the statement fails to parse, is a postgres source without @@ -887,8 +932,8 @@ fn parse_postgres_source_details<'a>(a: &'a str) -> Result { /// Extracts kafka source configuration (topic, group id prefix, connection /// id) from a catalog `create_sql`. Returns: /// -/// - jsonb `{"topic": , "group_id_prefix": , "connection_id": }` -/// for `CREATE SOURCE ... FROM KAFKA CONNECTION ... (TOPIC = ..., [GROUP ID PREFIX = ...])` +/// - jsonb `{"topic": , "group_id_prefix": , "connection_id": }` for +/// `CREATE SOURCE ... FROM KAFKA CONNECTION ... (TOPIC = ..., [GROUP ID PREFIX = ...])` /// statements. /// - jsonb `null` for any other statement. /// @@ -2327,4 +2372,70 @@ mod tests { Err("unresolved item name".to_string()) ); } + + // --- parse_catalog_item_references ---------------------------------------- + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_view() { + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + SELECT \"pg_catalog\".\"abs\"(\"a\"::[s20 AS \"pg_catalog\".\"int4\"]) \ + FROM [u1 AS \"materialize\".\"public\".\"t\"]"; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!( + out, + json!({ + "ids": ["s20", "u1"], + "named_funcs": [{"schema": "pg_catalog", "name": "abs"}], + "named_types": [], + "named_relations": [], + }) + ); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_connection() { + let sql = "CREATE CONNECTION \"materialize\".\"public\".\"kc\" TO KAFKA (\ + BROKER 'kafka:9092', \ + SSH TUNNEL [u5 AS \"materialize\".\"public\".\"ssh\"], \ + SASL PASSWORD = SECRET [u7 AS \"materialize\".\"public\".\"pw\"])"; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!(out["ids"], json!(["u5", "u7"])); + assert_eq!(out["named_funcs"], json!([])); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_cte_shadowing() { + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + WITH \"c\" AS (SELECT * FROM [u1 AS \"materialize\".\"public\".\"t\"]) \ + SELECT * FROM \"c\""; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!(out["ids"], json!(["u1"])); + assert_eq!(out["named_relations"], json!([])); + } + + #[mz_ore::test] + #[cfg_attr(miri, ignore)] // error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux` + fn item_references_name_based_relation_surfaces() { + // Stored SQL always prints relations as id references. A name-based + // relation must surface in "named_relations" rather than vanish. + let sql = "CREATE VIEW \"materialize\".\"public\".\"v\" AS \ + SELECT * FROM \"mz_catalog\".\"mz_tables\""; + let out = as_serde(super::parse_catalog_item_references(sql).expect("ok")); + assert_eq!( + out["named_relations"], + json!([{"schema": "mz_catalog", "name": "mz_tables"}]) + ); + } + + #[mz_ore::test] + fn item_references_parse_error() { + let err = super::parse_catalog_item_references("NOT SQL").unwrap_err(); + assert!( + matches!(err, EvalError::InvalidCatalogJson(msg) if msg.contains("failed to parse")), + "wrong error variant/message" + ); + } } diff --git a/src/expr/src/scalar/func/unary.rs b/src/expr/src/scalar/func/unary.rs index 4607716af19c5..c53f9d8ec60ef 100644 --- a/src/expr/src/scalar/func/unary.rs +++ b/src/expr/src/scalar/func/unary.rs @@ -467,6 +467,7 @@ derive_unary!( ParseCatalogAuditLogDetails, ParseCatalogCreateSql, ParseCatalogId, + ParseCatalogItemReferences, ParseCatalogPrivileges, ParseConnectionDetails, ParseKafkaSourceDetails, diff --git a/src/pgrepr-consts/src/oid.rs b/src/pgrepr-consts/src/oid.rs index 25eb0d8f7c98b..d768fbba66338 100644 --- a/src/pgrepr-consts/src/oid.rs +++ b/src/pgrepr-consts/src/oid.rs @@ -420,7 +420,7 @@ pub const MV_MZ_KAFKA_SINKS_OID: u32 = 16694; pub const MV_MZ_KAFKA_CONNECTIONS_OID: u32 = 16695; pub const MV_MZ_KAFKA_SOURCES_OID: u32 = 16696; pub const MV_MZ_POSTGRES_SOURCES_OID: u32 = 16697; -pub const TABLE_MZ_OBJECT_DEPENDENCIES_OID: u32 = 16698; +pub const MV_MZ_OBJECT_DEPENDENCIES_OID: u32 = 16698; pub const SOURCE_MZ_COMPUTE_DEPENDENCIES_OID: u32 = 16699; pub const SOURCE_MZ_COMPUTE_HYDRATION_STATUSES_OID: u32 = 16700; pub const LOG_MZ_COMPUTE_OPERATOR_HYDRATION_STATUSES_PER_WORKER_OID: u32 = 16701; @@ -831,3 +831,5 @@ pub const INDEX_MZ_METRIC_SINKS_IND_OID: u32 = 17121; pub const TABLE_MZ_OBJECT_HYDRATION_HISTORY_OID: u32 = 17122; pub const LOG_MZ_CLUSTER_REPLICA_RESOURCE_USAGE_OID: u32 = 17123; pub const TABLE_MZ_REPLICA_HYDRATION_HISTORY_OID: u32 = 17124; +pub const FUNC_PARSE_CATALOG_ITEM_REFERENCES_OID: u32 = 17125; +pub const VIEW_MZ_OBJECT_DEPENDENCIES_RAW_OID: u32 = 17126; diff --git a/src/pgrepr-consts/src/regproc.rs b/src/pgrepr-consts/src/regproc.rs index d3ad9c9c76674..63cc28e2ad5e2 100644 --- a/src/pgrepr-consts/src/regproc.rs +++ b/src/pgrepr-consts/src/regproc.rs @@ -685,6 +685,7 @@ pub const NAMES: &[(u32, &str)] = &[ (17113, "mz_aws_account_id"), (17114, "mz_aws_external_id_prefix"), (17115, "mz_aws_connection_role_arn"), + (17125, "mz_internal.parse_catalog_item_references"), // END GENERATED ]; diff --git a/src/sql-parser/src/ast.rs b/src/sql-parser/src/ast.rs index 268edd724fec7..85124990fcb66 100644 --- a/src/sql-parser/src/ast.rs +++ b/src/sql-parser/src/ast.rs @@ -23,6 +23,7 @@ #[macro_use] pub mod display; pub mod fold; +pub mod item_refs; pub mod visit; pub mod visit_mut; diff --git a/src/sql-parser/src/ast/item_refs.rs b/src/sql-parser/src/ast/item_refs.rs new file mode 100644 index 0000000000000..aa76b97be4bea --- /dev/null +++ b/src/sql-parser/src/ast/item_refs.rs @@ -0,0 +1,389 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software shall be governed +// by the Apache License, Version 2.0. + +//! Extraction of catalog item references from a raw SQL AST. + +use std::collections::BTreeSet; + +use crate::ast::visit::Visit; +use crate::ast::{ + CteBlock, Function, Ident, Query, Raw, RawDataType, RawItemName, Statement, UnresolvedItemName, +}; + +/// The catalog item references appearing in a statement. +/// +/// User statements are normalized and have IDs in its relation and type references (e.g. `[u1 AS +/// db.schema.name]`). For references that don't have IDs, we create separate `named_*` buckets that +/// contain their qualified names. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct ItemReferences { + /// Catalog IDs from `RawItemName::Id` references + pub ids: BTreeSet, + /// Named relations that don't carry an ID. Most commonly seen in builtin statements since they + /// are not normalized. CTE names are excluded: a name bound by a WITH clause refers to the + /// query-local binding rather than a catalog item, so recording it would fabricate an edge to + /// any catalog object that happens to share the name. + pub named_relations: BTreeSet, + pub named_funcs: BTreeSet, + pub named_types: BTreeSet, + /// Data-type references written `T[]`. These are + /// reported separately from `named_types` since `T[]` maps to the named type `_T`. + pub named_array_elements: BTreeSet, +} + +/// Collects all catalog item references from a statement. +pub fn collect_item_references(stmt: &Statement) -> ItemReferences { + let mut collector = ReferenceCollector { + refs: ItemReferences::default(), + cte_names: Vec::new(), + }; + collector.visit_statement(stmt); + collector.refs +} + +struct ReferenceCollector { + refs: ItemReferences, + /// Stack of CTE names currently in scope. + cte_names: Vec, +} + +impl ReferenceCollector { + fn record(&mut self, name: &RawItemName, position: Position) { + match name { + RawItemName::Id(id, _, _) => { + self.refs.ids.insert(id.clone()); + } + RawItemName::Name(name) => match position { + Position::Relation => { + // Filter out CTEs. + if let [only] = &name.0[..] { + if self.cte_names.contains(only) { + return; + } + } + self.refs.named_relations.insert(name.clone()); + } + Position::Func => { + self.refs.named_funcs.insert(name.clone()); + } + Position::Type => { + self.refs.named_types.insert(name.clone()); + } + }, + } + } +} + +enum Position { + Relation, + Func, + Type, +} + +impl<'ast> Visit<'ast, Raw> for ReferenceCollector { + fn visit_item_name(&mut self, node: &'ast RawItemName) { + self.record(node, Position::Relation); + } + + fn visit_function(&mut self, node: &'ast Function) { + let Function { + name, + args, + filter, + over, + distinct: _, + } = node; + self.record(name, Position::Func); + self.visit_function_args(args); + if let Some(filter) = filter { + self.visit_expr(filter); + } + if let Some(over) = over { + self.visit_window_spec(over); + } + } + + fn visit_data_type(&mut self, node: &'ast RawDataType) { + match node { + RawDataType::Array(elem_type) => match &**elem_type { + // `T[]` references the array type paired with `T`, not `T` + // itself. That pairing lives in `T`'s catalog details, which + // the parser cannot read, so the element is reported on its + // own field for the caller to map. + RawDataType::Other { + name: RawItemName::Name(name), + typ_mod: _, + } => { + self.refs.named_array_elements.insert(name.clone()); + } + // `[ AS ][]`. The id in hand names the element, and + // mapping it to the array type needs the catalog, so nothing + // is reported rather than an edge to the wrong type. Stored + // SQL never carries this spelling: name resolution prints a + // resolved array type as a single id reference. + RawDataType::Other { + name: RawItemName::Id(..), + typ_mod: _, + } => {} + // An array of a list, map, or array. Name resolution rejects + // the first two and the parser collapses `T[][]` into a 1D + // array, so none reach stored SQL. Recurse anyway so that no + // reference is dropped unreported. + elem_type => self.visit_data_type(elem_type), + }, + RawDataType::List(elem_type) => self.visit_data_type(elem_type), + RawDataType::Map { + key_type, + value_type, + } => { + self.visit_data_type(key_type); + self.visit_data_type(value_type); + } + RawDataType::Other { name, typ_mod: _ } => self.record(name, Position::Type), + } + } + + fn visit_query(&mut self, node: &'ast Query) { + let Query { + ctes, + body, + order_by, + limit, + offset, + } = node; + + let scope_depth = self.cte_names.len(); + // Keep track of CTEs such that we don't record them + // in `named_relations` + match ctes { + CteBlock::Simple(ctes) => { + for cte in ctes { + self.visit_cte(cte); + self.cte_names.push(cte.alias.name.clone()); + } + } + CteBlock::MutuallyRecursive(block) => { + for cte in &block.ctes { + self.cte_names.push(cte.name.clone()); + } + self.visit_mut_rec_block(block); + } + } + self.visit_set_expr(body); + for order_by in order_by { + self.visit_order_by_expr(order_by); + } + if let Some(limit) = limit { + self.visit_limit(limit); + } + if let Some(offset) = offset { + self.visit_expr(offset); + } + // Pop CTEs once visited. + self.cte_names.truncate(scope_depth); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::parser::parse_statements; + + #[track_caller] + fn collect(sql: &str) -> ItemReferences { + let stmts = parse_statements(sql).expect("valid sql"); + assert_eq!(stmts.len(), 1, "expected a single statement"); + collect_item_references(&stmts.into_iter().next().unwrap().ast) + } + + fn names(parts: &[&[&str]]) -> BTreeSet { + parts + .iter() + .map(|name| UnresolvedItemName(name.iter().map(|p| Ident::new_unchecked(*p)).collect())) + .collect() + } + + fn strings(items: &[&str]) -> BTreeSet { + items.iter().map(|s| s.to_string()).collect() + } + + #[mz_ore::test] + fn ids_from_bracketed_references() { + let refs = collect( + r#"CREATE VIEW v AS + SELECT a::[s20 AS "pg_catalog"."int4"] FROM [u1 AS "materialize"."public"."t"]"#, + ); + assert_eq!(refs.ids, strings(&["s20", "u1"])); + assert!(refs.named_relations.is_empty()); + assert!(refs.named_types.is_empty()); + } + + #[mz_ore::test] + fn versioned_id_reference() { + let refs = collect( + r#"CREATE VIEW v AS SELECT * FROM [u3 AS "materialize"."public"."t" VERSION 2]"#, + ); + assert_eq!(refs.ids, strings(&["u3"])); + } + + #[mz_ore::test] + fn function_names_classified() { + let refs = collect( + r#"CREATE VIEW v AS SELECT "pg_catalog"."abs"(1), count(*) FROM "mz_catalog"."mz_tables""#, + ); + assert_eq!( + refs.named_funcs, + names(&[&["pg_catalog", "abs"], &["count"]]) + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + } + + #[mz_ore::test] + fn function_args_still_visited() { + let refs = collect( + r#"CREATE VIEW v AS SELECT "pg_catalog"."abs"(a::"pg_catalog"."int8") + FROM "mz_catalog"."mz_tables""#, + ); + assert_eq!(refs.named_funcs, names(&[&["pg_catalog", "abs"]])); + assert_eq!(refs.named_types, names(&[&["pg_catalog", "int8"]])); + } + + #[mz_ore::test] + fn nested_data_types() { + let refs = collect( + r#"CREATE TABLE t (a "pg_catalog"."int4"[], b "pg_catalog"."text" list, + c map["pg_catalog"."text" => "pg_catalog"."int8" list])"#, + ); + // int4 sits under `[]`, so it is reported as an array element rather + // than as a type reference in its own right. + assert_eq!( + refs.named_types, + names(&[&["pg_catalog", "text"], &["pg_catalog", "int8"]]) + ); + assert_eq!(refs.named_array_elements, names(&[&["pg_catalog", "int4"]])); + } + + #[mz_ore::test] + fn array_nested_in_list() { + // A list of arrays still references the array type. + let refs = collect(r#"CREATE TABLE t (a "pg_catalog"."int4"[] list)"#); + assert_eq!(refs.named_array_elements, names(&[&["pg_catalog", "int4"]])); + assert!(refs.named_types.is_empty()); + } + + #[mz_ore::test] + fn array_of_id_reference_reports_nothing() { + // Mapping the element id to its array type needs the catalog, so + // nothing is reported rather than an edge to the element. + let refs = collect(r#"CREATE TABLE t (a [s20 AS "pg_catalog"."int4"][])"#); + assert!(refs.named_array_elements.is_empty()); + assert!(refs.named_types.is_empty()); + assert!(refs.ids.is_empty()); + } + + #[mz_ore::test] + fn array_of_list_still_visits_the_inner_type() { + // Name resolution rejects `T list[]`, but the collector must not drop + // the inner reference unreported. + let refs = collect(r#"CREATE TABLE t (a "pg_catalog"."int4" list[])"#); + assert_eq!(refs.named_types, names(&[&["pg_catalog", "int4"]])); + assert!(refs.named_array_elements.is_empty()); + } + + #[mz_ore::test] + fn cte_names_excluded() { + let refs = collect( + r#"CREATE VIEW v AS + WITH c AS (SELECT * FROM "mz_catalog"."mz_tables") + SELECT * FROM c JOIN "mz_catalog"."mz_views" ON true"#, + ); + assert_eq!( + refs.named_relations, + names(&[&["mz_catalog", "mz_tables"], &["mz_catalog", "mz_views"]]) + ); + } + + #[mz_ore::test] + fn mut_rec_cte_names_excluded() { + let refs = collect( + r#"CREATE VIEW v AS + WITH MUTUALLY RECURSIVE + reach (a int) AS (SELECT a FROM "mz_catalog"."mz_tables", reach) + SELECT * FROM reach"#, + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + // The int column type on the WMR CTE is still a type reference. + assert_eq!(refs.named_types, names(&[&["int4"]])); + } + + #[mz_ore::test] + fn qualified_cte_lookalike_not_excluded() { + // Only 1-part references can refer to a CTE. + let refs = collect( + r#"CREATE VIEW v AS + WITH mz_tables AS (SELECT 1 AS a) + SELECT * FROM mz_tables JOIN "mz_catalog"."mz_tables" ON true"#, + ); + assert_eq!(refs.named_relations, names(&[&["mz_catalog", "mz_tables"]])); + } + + #[mz_ore::test] + fn cte_scoping_is_lexical() { + // The CTE binding is scoped to the subquery that declares it: the + // same name in a sibling scope is a real relation reference. + let refs = collect( + r#"CREATE VIEW v AS + SELECT * FROM (WITH tbl AS (SELECT 1 AS a) SELECT * FROM tbl) x + JOIN tbl ON true"#, + ); + assert_eq!(refs.named_relations, names(&[&["tbl"]])); + } + + #[mz_ore::test] + fn simple_cte_definition_does_not_see_itself() { + // Mirroring the name resolver: a plain WITH binds a CTE's name only + // after its own definition, so the inner `tbl` is a real relation. + let refs = collect( + r#"CREATE VIEW v AS + WITH tbl AS (SELECT * FROM tbl) SELECT * FROM tbl"#, + ); + assert_eq!(refs.named_relations, names(&[&["tbl"]])); + } + + #[mz_ore::test] + fn later_simple_cte_sees_earlier() { + let refs = collect( + r#"CREATE VIEW v AS + WITH a AS (SELECT 1 AS x), b AS (SELECT * FROM a) + SELECT * FROM b"#, + ); + assert!(refs.named_relations.is_empty()); + } + + #[mz_ore::test] + fn connection_and_secret_references() { + let refs = collect( + r#"CREATE CONNECTION kc TO KAFKA ( + BROKER 'kafka:9092', + SSH TUNNEL [u5 AS "materialize"."public"."ssh"], + SASL PASSWORD = SECRET [u7 AS "materialize"."public"."pw"] + )"#, + ); + assert_eq!(refs.ids, strings(&["u5", "u7"])); + } + + #[mz_ore::test] + fn index_references() { + let refs = collect( + r#"CREATE INDEX i IN CLUSTER [u1] ON [u2 AS "materialize"."public"."t"] ("pg_catalog"."abs"(a))"#, + ); + assert_eq!(refs.ids, strings(&["u2"])); + assert_eq!(refs.named_funcs, names(&[&["pg_catalog", "abs"]])); + } +} diff --git a/src/sql/src/func.rs b/src/sql/src/func.rs index 8ef18c0c19c69..3cad3f2269471 100644 --- a/src/sql/src/func.rs +++ b/src/sql/src/func.rs @@ -5355,6 +5355,11 @@ pub static MZ_INTERNAL_BUILTINS: LazyLock> = LazyLo params!(Jsonb) => UnaryFunc::ParseCatalogId(func::ParseCatalogId) => String, oid::FUNC_PARSE_CATALOG_ID_OID; }, + "parse_catalog_item_references" => Scalar { + params!(String) => UnaryFunc::ParseCatalogItemReferences( + func::ParseCatalogItemReferences, + ) => Jsonb, oid::FUNC_PARSE_CATALOG_ITEM_REFERENCES_OID; + }, "parse_catalog_privileges" => Scalar { params!(Jsonb) => UnaryFunc::ParseCatalogPrivileges(func::ParseCatalogPrivileges) => SqlScalarType::Array(Box::new(SqlScalarType::MzAclItem)), diff --git a/src/sql/src/names.rs b/src/sql/src/names.rs index 3c50e68b255fb..1fd703a9d8885 100644 --- a/src/sql/src/names.rs +++ b/src/sql/src/names.rs @@ -40,7 +40,7 @@ use crate::ast::{ RawDataType, RawItemName, Statement, UnresolvedItemName, UnresolvedObjectName, }; use crate::catalog::{ - CatalogError, CatalogItem, CatalogItemType, CatalogType, CatalogTypeDetails, SessionCatalog, + CatalogError, CatalogItem, CatalogItemType, CatalogTypeDetails, SessionCatalog, }; use crate::normalize; use crate::plan::PlanError; @@ -1355,10 +1355,21 @@ impl<'a> NameResolver<'a> { } fn resolve_data_type(&mut self, data_type: RawDataType) -> Result { + self.resolve_data_type_inner(data_type, true) + } + + fn resolve_data_type_inner( + &mut self, + data_type: RawDataType, + should_insert_id: bool, + ) -> Result { match data_type { RawDataType::Array(elem_type) => { let name = elem_type.to_string(); - match self.resolve_data_type(*elem_type)? { + // `T[]` recursively resolves to its element type, but we + // only want to insert its array type `_T` and not its element + // type `T`. Thus we set `should_insert_id` to `false`. + match self.resolve_data_type_inner(*elem_type, false)? { ResolvedDataType::AnonymousList(_) | ResolvedDataType::AnonymousMap { .. } => { sql_bail!("type \"{}[]\" does not exist", name) } @@ -1384,6 +1395,11 @@ impl<'a> NameResolver<'a> { ); } }; + + // We don't need to check `should_insert_id` here since we're + // guaranteed to insert just one array item type for an + // array. For multi-dimensional arrays, the parser always + // collapses it into a 1D array. self.ids.insert(array_item.id(), BTreeSet::new()); Ok(ResolvedDataType::Named { id: array_item.id(), @@ -1397,15 +1413,15 @@ impl<'a> NameResolver<'a> { } } RawDataType::List(elem_type) => { - let elem_type = self.resolve_data_type(*elem_type)?; + let elem_type = self.resolve_data_type_inner(*elem_type, should_insert_id)?; Ok(ResolvedDataType::AnonymousList(Box::new(elem_type))) } RawDataType::Map { key_type, value_type, } => { - let key_type = self.resolve_data_type(*key_type)?; - let value_type = self.resolve_data_type(*value_type)?; + let key_type = self.resolve_data_type_inner(*key_type, should_insert_id)?; + let value_type = self.resolve_data_type_inner(*value_type, should_insert_id)?; Ok(ResolvedDataType::AnonymousMap { key_type: Box::new(key_type), value_type: Box::new(value_type), @@ -1433,17 +1449,8 @@ impl<'a> NameResolver<'a> { (full_name, item) } }; - self.ids.insert(item.id(), BTreeSet::new()); - // If this is a named array type, then make sure to include the element reference - // in the resolved IDs. This helps ensure that named array types are resolved the - // same as an array type with the same element type. For example, `int4[]` and - // `_int4` should have the same set of resolved IDs. - if let Some(CatalogTypeDetails { - typ: CatalogType::Array { element_reference }, - .. - }) = item.type_details() - { - self.ids.insert(*element_reference, BTreeSet::new()); + if should_insert_id { + self.ids.insert(item.id(), BTreeSet::new()); } Ok(ResolvedDataType::Named { id: item.id(), diff --git a/src/storage/src/sink/iceberg.rs b/src/storage/src/sink/iceberg.rs index c416e450ee33d..eb6526aea0ddf 100644 --- a/src/storage/src/sink/iceberg.rs +++ b/src/storage/src/sink/iceberg.rs @@ -927,6 +927,20 @@ async fn load_or_create_table( Ok(table) => { // Table exists, return it // TODO: Add proper schema evolution/validation to ensure compatibility + let current_schema = table.metadata().current_schema(); + if !(current_schema.as_struct().eq(schema.as_struct()) + && current_schema + .identifier_field_ids() + .eq(schema.identifier_field_ids())) + { + anyhow::bail!( + "Iceberg table '{}' schema does not match expected schema. \ + Current schema: {:?}, expected schema: {:?}", + table_name, + current_schema, + schema + ); + } Ok(table) } Err(err) => { diff --git a/test/iceberg/alter-table-add-column.td b/test/iceberg/alter-table-add-column.td new file mode 100644 index 0000000000000..a9df7a3bbc8d4 --- /dev/null +++ b/test/iceberg/alter-table-add-column.td @@ -0,0 +1,96 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# An Iceberg sink whose input relation has more columns than the Iceberg table it +# writes to must not panic the storage worker. +# +# `load_or_create_table` hands back a pre-existing Iceberg table without checking +# that its schema still matches the sink's input, and the writer derives its Arrow +# column builders from that table schema (`merge_materialize_metadata_into_iceberg_schema`). +# Once the Materialize relation has gained a column, every row carries more datums +# than the builder has columns, and `ArrowBuilder::add_row` reaches +# `zip_eq` with iterators of different lengths: +# +# itertools: .zip_eq() reached end of one iterator before the other +# mz_arrow_util::builder::ArrowBuilder::add_row +# mz_storage::sink::iceberg::AppendEnvelopeHandler::row_to_batch +# mz_storage::sink::iceberg::write_data_files + +$ postgres-connect name=mz_system url=postgres://mz_system:materialize@${testdrive.materialize-internal-sql-addr} + +$ postgres-execute connection=mz_system +ALTER SYSTEM SET enable_alter_table_add_column = true + +> CREATE SECRET IF NOT EXISTS evolve_access_key_secret AS '${arg.s3-access-key}' + +> CREATE CONNECTION evolve_polaris TO ICEBERG CATALOG ( + CATALOG TYPE = 'REST', + URL = 'http://polaris:8181/api/catalog', + CREDENTIAL = 'root:root', + WAREHOUSE = 'default_catalog', + SCOPE = 'PRINCIPAL_ROLE:ALL' + ); + +> CREATE TABLE evolve_src (a int); + +> INSERT INTO evolve_src VALUES (1); + +# Creates the Iceberg table with columns (a, _mz_diff, _mz_timestamp). +> CREATE SINK evolve_sink_v1 + FROM evolve_src + INTO ICEBERG CATALOG CONNECTION evolve_polaris ( + NAMESPACE 'default_namespace', + TABLE 'evolve_table' + ) + MODE APPEND + WITH (COMMIT INTERVAL '1s'); + +# Iceberg sinks commit asynchronously; wait for the table to be created and the +# first batch committed before dropping the sink. +$ sleep-is-probably-flaky-i-have-justified-my-need-with-a-comment duration=5s + +$ duckdb-execute name=evolve_iceberg +CREATE SECRET s3_secret_evolve (TYPE S3, KEY_ID 'tduser', SECRET '${arg.s3-access-key}', ENDPOINT '${arg.aws-endpoint}', URL_STYLE 'path', USE_SSL false, REGION 'minio'); +SET unsafe_enable_version_guessing = true; + +$ duckdb-query name=evolve_iceberg +SELECT a, _mz_diff FROM iceberg_scan('s3://test-bucket/default_namespace/evolve_table') ORDER BY a +1 1 + +> DROP SINK evolve_sink_v1; + +> ALTER TABLE evolve_src ADD COLUMN b text; + +# The new sink's input has two columns, the Iceberg table still has one. +> CREATE SINK evolve_sink_v2 + FROM evolve_src + INTO ICEBERG CATALOG CONNECTION evolve_polaris ( + NAMESPACE 'default_namespace', + TABLE 'evolve_table' + ) + MODE APPEND + WITH (COMMIT INTERVAL '1s'); + +> INSERT INTO evolve_src VALUES (2, 'two'); + +$ sleep-is-probably-flaky-i-have-justified-my-need-with-a-comment duration=10s + +# The sink must not have taken the process down with it. Any outcome that keeps +# the storage worker alive (an error status, or a schema evolution that lets the +# write through) is acceptable here, so only assert that we can still talk to the +# cluster hosting the sink. +> SELECT count(*) FROM evolve_src; +2 + +> SELECT count(*) > 0 + FROM mz_internal.mz_sink_status_history h + JOIN mz_sinks s ON s.id = h.sink_id + WHERE s.name = 'evolve_sink_v2' + AND h.error LIKE '%schema does not match expected schema%'; +true diff --git a/test/iceberg/mzcompose.py b/test/iceberg/mzcompose.py index 37f1bb460a22e..d0da220de564e 100644 --- a/test/iceberg/mzcompose.py +++ b/test/iceberg/mzcompose.py @@ -204,6 +204,26 @@ def workflow_finite_source(c: Composition) -> None: ) +def workflow_alter_table_add_column(c: Composition) -> None: + """A sink pointed at an Iceberg table whose schema is narrower than the + sink's input relation must not panic the storage worker. The writer builds + its Arrow column builders from the Iceberg table's schema, so a row with + more datums than that schema has columns reaches `zip_eq` in + `ArrowBuilder::add_row`.""" + key = _setup(c) + + c.run_testdrive_files( + f"--var=s3-access-key={key}", + "--var=aws-endpoint=minio:9000", + "alter-table-add-column.td", + ) + + logs = c.invoke("logs", "materialized", capture=True) + assert ( + "zip_eq" not in logs.stdout + ), "storage worker panicked in ArrowBuilder::add_row" + + def _polaris_get(table_url: str, access_token: str) -> dict: """GET table metadata from Polaris REST API (always returns latest).""" req = urllib.request.Request( diff --git a/test/sqllogictest/autogenerated/mz_internal.slt b/test/sqllogictest/autogenerated/mz_internal.slt index 62a3ee04f6785..66cd3a52e3a4e 100644 --- a/test/sqllogictest/autogenerated/mz_internal.slt +++ b/test/sqllogictest/autogenerated/mz_internal.slt @@ -859,6 +859,7 @@ mz_notices_redacted mz_object_arrangement_size_history mz_object_arrangement_sizes mz_object_dependencies +mz_object_dependencies_raw mz_object_fully_qualified_names mz_object_global_ids mz_object_graph_edges diff --git a/test/sqllogictest/catalog_server_explain.slt b/test/sqllogictest/catalog_server_explain.slt index 54e9301255e67..4fb1faa189f0a 100644 --- a/test/sqllogictest/catalog_server_explain.slt +++ b/test/sqllogictest/catalog_server_explain.slt @@ -5127,7 +5127,7 @@ mz_catalog.mz_materialized_views: Project: #4, #0, #5, #1, #6, #2, #9, #8, #3, #7 Map: {=r/s1, s1=r/s1}, "s1" →Arrange (#1{schema_name}, #2{name}) - →Constant (43 rows) + →Constant (44 rows) →Arrange (empty key) (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #4, #3, #5 @@ -5472,7 +5472,7 @@ mz_catalog.mz_tables: Project: #3, #0, #4, #1, #5, #2, #6, #6, #6 Map: "s1", null →Arrange (#1{schema_name}, #2{name}) - →Constant (31 rows) + →Constant (30 rows) →Arrange (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #5, #4, #6 @@ -5527,7 +5527,7 @@ mz_catalog.mz_views: Project: #5, #0, #6, #1, #2, #8, #3, #4, #7 Map: "s1" →Arrange (#1{schema_name}, #2{name}) - →Constant (193 rows) + →Constant (194 rows) →Arrange (#0{schema_name}) (#0{schema_name}, #1{name}) →Fused with Child Map/Filter/Project Project: #5, #4, #6 @@ -5815,6 +5815,116 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_object_dependencies"; +---- +mz_internal.mz_object_dependencies: + →With + cte l0 = + →Fused with Child Map/Filter/Project + Project: #3, #4 + Filter: ("Item" = (#0{data} ->> "kind")) AND ((#2 ->> "ephemeral_owner_session")) IS NULL + Map: (#0{data} -> "value"), parse_catalog_id((#1 -> "gid")), parse_catalog_item_references((((#2 -> "definition") -> "V1") ->> "create_sql")) + →Read mz_internal.mz_catalog_raw + cte l1 = + →Fused with Child Map/Filter/Project + Project: #0..=#4 + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Return + →Distinct GroupAggregate + →Union + →Fused with Child Map/Filter/Project + Project: #0, #2 + →Table Function jsonb_array_elements_text((#1{refs} -> "ids")) + →Stream l0 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{func} ->> "name"), (#1{func} ->> "schema")] + →Arrange ((#1{func} ->> "name"), (#1{func} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{func} ->> "name")) IS NOT NULL AND ((#2{func} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_funcs")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("8" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{typ} ->> "name"), (#1{typ} ->> "schema")] + →Arrange ((#1{typ} ->> "name"), (#1{typ} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{typ} ->> "name")) IS NOT NULL AND ((#2{typ} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_types")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("7" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:mz_catalog_raw[#2{object_name}, #1{schema_name}] » %0[(#1{rel} ->> "name"), (#1{rel} ->> "schema")] + →Arrange ((#1{rel} ->> "name"), (#1{rel} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{rel} ->> "name")) IS NOT NULL AND ((#2{rel} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_relations")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2 != "7") AND (#2 != "8") AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_type"), (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Delta Join [%0[#0{object_schema}..=#2{object_type}] » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] [%1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %0[#0{object_schema}..=#2{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] [%2:mz_catalog_raw[#1{schema_name}, #2{object_name}] » %0[#3{ref_schema}, #4{ref_name}] » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}]] + path %0: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + path %1: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + path %2: + after %0: + Project: #4..=#6, #2 + Filter: case when (#7{ref_kind} = "func") then (#3{object_type} = "8") else case when (#7{ref_kind} = "type") then (#3{object_type} = "7") else ((#3{object_type} != "7") AND (#3{object_type} != "8")) end end + →Arrange (#0{object_schema}..=#2{object_type}) (#3{ref_schema}, #4{ref_name}) + →Constant (1024 rows) + →Arrange (#1{schema_name}..=#3{object_type}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #2, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "object_type"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Arrange (#1{schema_name}, #2{object_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2, #5 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")), (#1 ->> "object_type") + →Read mz_internal.mz_catalog_raw + →Differential Join %0:mz_catalog_raw[((#0{data} -> "key") ->> "name")] » %1:mz_catalog_raw[((#0{data} -> "key") ->> "object_name")] + after %1: + Project: #3, #4 + Map: ("si" || ((#1{data} -> "value") ->> "catalog_id")), ("s" || ((#2{data} -> "value") ->> "catalog_id")) + →Arrange (((#0{data} -> "key") ->> "name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("ClusterIntrospectionSourceIndex" = (#0{data} ->> "kind")) AND ((#1 ->> "name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arrange (((#0{data} -> "key") ->> "object_name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("2" = (#1 ->> "object_type")) AND ("mz_introspection" = (#1 ->> "schema_name")) AND ((#1 ->> "object_name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + +Source mz_internal.mz_catalog_raw + project=(#0, #1) + map=((#0{data} -> "key")) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN MATERIALIZED VIEW "mz_internal"."mz_overridden_system_parameters"; ---- @@ -8226,7 +8336,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_materialized_views"; ---- Explained Query (fast path): - →Constant (43 rows) + →Constant (44 rows) Target cluster: mz_catalog_server @@ -8246,7 +8356,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_tables"; ---- Explained Query (fast path): - →Constant (31 rows) + →Constant (30 rows) Target cluster: mz_catalog_server @@ -8256,7 +8366,7 @@ query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_builtin_views"; ---- Explained Query (fast path): - →Constant (193 rows) + →Constant (194 rows) Target cluster: mz_catalog_server @@ -10108,6 +10218,108 @@ Target cluster: mz_catalog_server EOF +query T multiline +EXPLAIN SELECT * FROM "mz_internal"."mz_object_dependencies_raw"; +---- +Explained Query: + →With + cte l0 = + →Fused with Child Map/Filter/Project + Project: #3, #4 + Filter: ("Item" = (#0{data} ->> "kind")) AND ((#2 ->> "ephemeral_owner_session")) IS NULL + Map: (#0{data} -> "value"), parse_catalog_id((#1 -> "gid")), parse_catalog_item_references((((#2 -> "definition") -> "V1") ->> "create_sql")) + →Read mz_internal.mz_catalog_raw + cte l1 = + →Fused with Child Map/Filter/Project + Project: #0..=#4 + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Return + →Distinct GroupAggregate + →Union + →Fused with Child Map/Filter/Project + Project: #0, #2 + →Table Function jsonb_array_elements_text((#1{refs} -> "ids")) + →Stream l0 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{func} ->> "name"), (#1{func} ->> "schema")] + →Arrange ((#1{func} ->> "name"), (#1{func} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{func} ->> "name")) IS NOT NULL AND ((#2{func} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_funcs")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("8" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:l1[#2{object_name}, #1{schema_name}] » %0[(#1{typ} ->> "name"), (#1{typ} ->> "schema")] + →Arrange ((#1{typ} ->> "name"), (#1{typ} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{typ} ->> "name")) IS NOT NULL AND ((#2{typ} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_types")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("7" = (#1 ->> "object_type")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + →Read l1 + →Differential Join %1:mz_catalog_raw[#2{object_name}, #1{schema_name}] » %0[(#1{rel} ->> "name"), (#1{rel} ->> "schema")] + →Arrange ((#1{rel} ->> "name"), (#1{rel} ->> "schema")) + →Fused with Child Map/Filter/Project + Project: #0, #2 + Filter: ((#2{rel} ->> "name")) IS NOT NULL AND ((#2{rel} ->> "schema")) IS NOT NULL + →Table Function jsonb_array_elements((#1{refs} -> "named_relations")) + →Stream l0 + →Arrange (#2{object_name}, #1{schema_name}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2 != "7") AND (#2 != "8") AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_type"), (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →One-Shot Delta Join [%0 » %1:mz_catalog_raw[#1{schema_name}..=#3{object_type}] » %2:mz_catalog_raw[#1{schema_name}, #2{object_name}]] + path %0: + after %2: + Project: #3, #4 + Filter: case when (#2{ref_kind} = "func") then (#5{object_type} = "8") else case when (#2{ref_kind} = "type") then (#5{object_type} = "7") else ((#5{object_type} != "7") AND (#5{object_type} != "8")) end end + →Unarranged Raw Stream + →Constant (1024 rows) + →Arrange (#1{schema_name}..=#3{object_type}) + →Fused with Child Map/Filter/Project + Project: #5, #4, #2, #3 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL AND (#4) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "object_type"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")) + →Read mz_internal.mz_catalog_raw + →Arrange (#1{schema_name}, #2{object_name}) + →Fused with Child Map/Filter/Project + Project: #4, #3, #2, #5 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND (#2) IS NOT NULL AND (#3) IS NOT NULL + Map: (#1 ->> "object_name"), (#1 ->> "schema_name"), ("s" || ((#0{data} -> "value") ->> "catalog_id")), (#1 ->> "object_type") + →Read mz_internal.mz_catalog_raw + →Differential Join %0:mz_catalog_raw[((#0{data} -> "key") ->> "name")] » %1:mz_catalog_raw[((#0{data} -> "key") ->> "object_name")] + after %1: + Project: #3, #4 + Map: ("si" || ((#1{data} -> "value") ->> "catalog_id")), ("s" || ((#2{data} -> "value") ->> "catalog_id")) + →Arrange (((#0{data} -> "key") ->> "name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("ClusterIntrospectionSourceIndex" = (#0{data} ->> "kind")) AND ((#1 ->> "name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + →Arrange (((#0{data} -> "key") ->> "object_name")) + →Fused with Child Map/Filter/Project + Project: #0 + Filter: ("GidMapping" = (#0{data} ->> "kind")) AND ("2" = (#1 ->> "object_type")) AND ("mz_introspection" = (#1 ->> "schema_name")) AND ((#1 ->> "object_name")) IS NOT NULL + →Read mz_internal.mz_catalog_raw + +Source mz_internal.mz_catalog_raw + project=(#0, #1) + map=((#0{data} -> "key")) + +Target cluster: mz_catalog_server + +EOF + query T multiline EXPLAIN SELECT * FROM "mz_internal"."mz_object_fully_qualified_names"; ---- diff --git a/test/sqllogictest/funcs.slt b/test/sqllogictest/funcs.slt index 96cd26a74403d..63f5c442fb789 100644 --- a/test/sqllogictest/funcs.slt +++ b/test/sqllogictest/funcs.slt @@ -1769,3 +1769,22 @@ SELECT textrecv() query error cannot call function with arguments of type internal SELECT textrecv('abc') + +# `mz_internal.parse_catalog_item_references` extracts the catalog item +# references from a stored `create_sql` string. + +query T +SELECT mz_internal.parse_catalog_item_references('CREATE VIEW "materialize"."public"."v" AS SELECT "pg_catalog"."abs"("a"::[s20 AS "pg_catalog"."int4"]) FROM [u1 AS "materialize"."public"."t"]') +---- +{"ids":["s20","u1"],"named_funcs":[{"name":"abs","schema":"pg_catalog"}],"named_relations":[],"named_types":[]} + +# Name-based relation references never appear in stored SQL, which prints +# relations as id references. Reporting them keeps a statement that violates +# that from silently losing its dependency edges. +query T +SELECT mz_internal.parse_catalog_item_references('CREATE VIEW "materialize"."public"."v" AS SELECT * FROM "mz_catalog"."mz_tables"') +---- +{"ids":[],"named_funcs":[],"named_relations":[{"name":"mz_tables","schema":"mz_catalog"}],"named_types":[]} + +query error invalid catalog JSON: failed to parse create_sql +SELECT mz_internal.parse_catalog_item_references('NOT SQL') diff --git a/test/sqllogictest/information_schema_tables.slt b/test/sqllogictest/information_schema_tables.slt index 881b0b0b89c44..5b27ee141814f 100644 --- a/test/sqllogictest/information_schema_tables.slt +++ b/test/sqllogictest/information_schema_tables.slt @@ -494,7 +494,11 @@ SOURCE materialize mz_internal mz_object_dependencies -BASE TABLE +MATERIALIZED VIEW +materialize +mz_internal +mz_object_dependencies_raw +VIEW materialize mz_internal mz_object_fully_qualified_names diff --git a/test/sqllogictest/mz_catalog_server_index_accounting.slt b/test/sqllogictest/mz_catalog_server_index_accounting.slt index 05c34d29a228c..75b12a7d85855 100644 --- a/test/sqllogictest/mz_catalog_server_index_accounting.slt +++ b/test/sqllogictest/mz_catalog_server_index_accounting.slt @@ -37,110 +37,110 @@ mz_arrangement_heap_capacity_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangemen mz_arrangement_heap_size_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_heap_size_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_heap_size_raw"␠("operator_id",␠"worker_id") mz_arrangement_records_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_records_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_records_raw"␠("operator_id",␠"worker_id") mz_arrangement_sharing_raw_s2_primary_idx CREATE␠INDEX␠"mz_arrangement_sharing_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_arrangement_sharing_raw"␠("operator_id",␠"worker_id") -mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s521␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") -mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s770␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") +mz_cluster_auto_scaling_strategies_ind CREATE␠INDEX␠"mz_cluster_auto_scaling_strategies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s523␠AS␠"mz_internal"."mz_cluster_auto_scaling_strategies"]␠("cluster_id") +mz_cluster_deployment_lineage_ind CREATE␠INDEX␠"mz_cluster_deployment_lineage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s772␠AS␠"mz_internal"."mz_cluster_deployment_lineage"]␠("cluster_id") mz_cluster_prometheus_metrics_s2_primary_idx CREATE␠INDEX␠"mz_cluster_prometheus_metrics_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_cluster_prometheus_metrics"␠("process_id",␠"metric_name",␠"labels") -mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s520␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") -mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s764␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") -mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s619␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") -mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s526␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") -mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s527␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") -mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s620␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") +mz_cluster_reconfigurations_ind CREATE␠INDEX␠"mz_cluster_reconfigurations_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s522␠AS␠"mz_internal"."mz_cluster_reconfigurations"]␠("cluster_id") +mz_cluster_replica_frontiers_ind CREATE␠INDEX␠"mz_cluster_replica_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s766␠AS␠"mz_catalog"."mz_cluster_replica_frontiers"]␠("object_id") +mz_cluster_replica_history_ind CREATE␠INDEX␠"mz_cluster_replica_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s621␠AS␠"mz_internal"."mz_cluster_replica_history"]␠("dropped_at") +mz_cluster_replica_metrics_history_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s528␠AS␠"mz_internal"."mz_cluster_replica_metrics_history"]␠("replica_id") +mz_cluster_replica_metrics_ind CREATE␠INDEX␠"mz_cluster_replica_metrics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s529␠AS␠"mz_internal"."mz_cluster_replica_metrics"]␠("replica_id") +mz_cluster_replica_name_history_ind CREATE␠INDEX␠"mz_cluster_replica_name_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s622␠AS␠"mz_internal"."mz_cluster_replica_name_history"]␠("id") mz_cluster_replica_resource_usage_s2_primary_idx CREATE␠INDEX␠"mz_cluster_replica_resource_usage_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_cluster_replica_resource_usage"␠("process_id",␠"source",␠"metric") -mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s516␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") -mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s515␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") -mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s528␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") -mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s529␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") -mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s525␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") -mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s517␠AS␠"mz_catalog"."mz_clusters"]␠("id") -mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s488␠AS␠"mz_catalog"."mz_columns"]␠("name") -mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s544␠AS␠"mz_internal"."mz_comments"]␠("id") +mz_cluster_replica_size_internal_ind CREATE␠INDEX␠"mz_cluster_replica_size_internal_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s518␠AS␠"mz_internal"."mz_cluster_replica_size_internal"]␠("size") +mz_cluster_replica_sizes_ind CREATE␠INDEX␠"mz_cluster_replica_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s517␠AS␠"mz_catalog"."mz_cluster_replica_sizes"]␠("size") +mz_cluster_replica_status_history_ind CREATE␠INDEX␠"mz_cluster_replica_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s530␠AS␠"mz_internal"."mz_cluster_replica_status_history"]␠("replica_id") +mz_cluster_replica_statuses_ind CREATE␠INDEX␠"mz_cluster_replica_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s531␠AS␠"mz_internal"."mz_cluster_replica_statuses"]␠("replica_id") +mz_cluster_replicas_ind CREATE␠INDEX␠"mz_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s527␠AS␠"mz_catalog"."mz_cluster_replicas"]␠("id") +mz_clusters_ind CREATE␠INDEX␠"mz_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s519␠AS␠"mz_catalog"."mz_clusters"]␠("id") +mz_columns_ind CREATE␠INDEX␠"mz_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s490␠AS␠"mz_catalog"."mz_columns"]␠("name") +mz_comments_ind CREATE␠INDEX␠"mz_comments_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s546␠AS␠"mz_internal"."mz_comments"]␠("id") mz_compute_dataflow_global_ids_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_dataflow_global_ids_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_dataflow_global_ids_per_worker"␠("id",␠"worker_id",␠"global_id") -mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s745␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") +mz_compute_dependencies_ind CREATE␠INDEX␠"mz_compute_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s747␠AS␠"mz_internal"."mz_compute_dependencies"]␠("dependency_id") mz_compute_error_counts_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_error_counts_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_error_counts_raw"␠("export_id",␠"worker_id") mz_compute_exports_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_exports_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_exports_per_worker"␠("export_id",␠"worker_id") mz_compute_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_frontiers_per_worker"␠("export_id",␠"worker_id") -mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s755␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") +mz_compute_hydration_times_ind CREATE␠INDEX␠"mz_compute_hydration_times_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s757␠AS␠"mz_internal"."mz_compute_hydration_times"]␠("replica_id") mz_compute_hydration_times_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_hydration_times_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_hydration_times_per_worker"␠("export_id",␠"worker_id") mz_compute_import_frontiers_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_import_frontiers_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_import_frontiers_per_worker"␠("export_id",␠"import_id",␠"worker_id") mz_compute_lir_mapping_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_lir_mapping_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_lir_mapping_per_worker"␠("global_id",␠"lir_id",␠"worker_id") mz_compute_operator_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_durations_histogram_raw"␠("id",␠"worker_id",␠"duration_ns") mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_compute_operator_hydration_statuses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_compute_operator_hydration_statuses_per_worker"␠("export_id",␠"lir_id",␠"worker_id") -mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s523␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") -mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s751␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") -mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s750␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") -mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") -mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s486␠AS␠"mz_catalog"."mz_databases"]␠("name") +mz_connections_ind CREATE␠INDEX␠"mz_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s525␠AS␠"mz_catalog"."mz_connections"]␠("schema_id") +mz_console_cluster_utilization_overview_24h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_24h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s753␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_24h"]␠("cluster_id") +mz_console_cluster_utilization_overview_3h_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_3h_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s752␠AS␠"mz_internal"."mz_console_cluster_utilization_overview_3h"]␠("cluster_id") +mz_console_cluster_utilization_overview_ind CREATE␠INDEX␠"mz_console_cluster_utilization_overview_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s751␠AS␠"mz_internal"."mz_console_cluster_utilization_overview"]␠("cluster_id") +mz_databases_ind CREATE␠INDEX␠"mz_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s488␠AS␠"mz_catalog"."mz_databases"]␠("name") mz_dataflow_addresses_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_addresses_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_addresses_per_worker"␠("id",␠"worker_id") mz_dataflow_channels_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_channels_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_channels_per_worker"␠("id",␠"worker_id") mz_dataflow_operator_reachability_raw_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operator_reachability_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operator_reachability_raw"␠("id",␠"worker_id",␠"source",␠"port",␠"update_type",␠"time") mz_dataflow_operators_per_worker_s2_primary_idx CREATE␠INDEX␠"mz_dataflow_operators_per_worker_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_dataflow_operators_per_worker"␠("id",␠"worker_id") -mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s736␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") -mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s766␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") -mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s489␠AS␠"mz_catalog"."mz_indexes"]␠("id") -mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s483␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") -mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s548␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") +mz_frontiers_ind CREATE␠INDEX␠"mz_frontiers_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s738␠AS␠"mz_internal"."mz_frontiers"]␠("object_id") +mz_hydration_statuses_ind CREATE␠INDEX␠"mz_hydration_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s768␠AS␠"mz_internal"."mz_hydration_statuses"]␠("object_id",␠"replica_id") +mz_indexes_ind CREATE␠INDEX␠"mz_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s491␠AS␠"mz_catalog"."mz_indexes"]␠("id") +mz_kafka_sources_ind CREATE␠INDEX␠"mz_kafka_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_catalog"."mz_kafka_sources"]␠("id") +mz_materialized_views_ind CREATE␠INDEX␠"mz_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s550␠AS␠"mz_catalog"."mz_materialized_views"]␠("id") mz_message_batch_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_batch_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_batch_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_batch_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_received_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_received_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_received_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") mz_message_counts_sent_raw_s2_primary_idx CREATE␠INDEX␠"mz_message_counts_sent_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_message_counts_sent_raw"␠("channel_id",␠"from_worker_id",␠"to_worker_id") -mz_metric_sinks_ind CREATE␠INDEX␠"mz_metric_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s546␠AS␠"mz_internal"."mz_metric_sinks"]␠("id") -mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s844␠AS␠"mz_internal"."mz_notices"]␠("id") -mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") -mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") -mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s756␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") -mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s484␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") -mz_object_graph_edges_ind CREATE␠INDEX␠"mz_object_graph_edges_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s747␠AS␠"mz_internal"."mz_object_graph_edges"]␠("object_id") -mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s559␠AS␠"mz_internal"."mz_object_history"]␠("id") -mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s560␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") -mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s576␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") -mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s556␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") +mz_metric_sinks_ind CREATE␠INDEX␠"mz_metric_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s548␠AS␠"mz_internal"."mz_metric_sinks"]␠("id") +mz_notices_ind CREATE␠INDEX␠"mz_notices_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s846␠AS␠"mz_internal"."mz_notices"]␠("id") +mz_object_arrangement_size_history_object_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_object_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s760␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("object_id") +mz_object_arrangement_size_history_ts_ind CREATE␠INDEX␠"mz_object_arrangement_size_history_ts_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s760␠AS␠"mz_internal"."mz_object_arrangement_size_history"]␠("collection_timestamp") +mz_object_arrangement_sizes_ind CREATE␠INDEX␠"mz_object_arrangement_sizes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s758␠AS␠"mz_internal"."mz_object_arrangement_sizes"]␠("replica_id") +mz_object_dependencies_ind CREATE␠INDEX␠"mz_object_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s486␠AS␠"mz_internal"."mz_object_dependencies"]␠("object_id") +mz_object_graph_edges_ind CREATE␠INDEX␠"mz_object_graph_edges_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s749␠AS␠"mz_internal"."mz_object_graph_edges"]␠("object_id") +mz_object_history_ind CREATE␠INDEX␠"mz_object_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s561␠AS␠"mz_internal"."mz_object_history"]␠("id") +mz_object_lifetimes_ind CREATE␠INDEX␠"mz_object_lifetimes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s562␠AS␠"mz_internal"."mz_object_lifetimes"]␠("id") +mz_object_transitive_dependencies_ind CREATE␠INDEX␠"mz_object_transitive_dependencies_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s578␠AS␠"mz_internal"."mz_object_transitive_dependencies"]␠("object_id") +mz_objects_ind CREATE␠INDEX␠"mz_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s558␠AS␠"mz_catalog"."mz_objects"]␠("schema_id") mz_peek_durations_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_peek_durations_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_peek_durations_histogram_raw"␠("worker_id",␠"type",␠"duration_ns") -mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s720␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") -mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s716␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") -mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s835␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") -mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s507␠AS␠"mz_catalog"."mz_roles"]␠("id") +mz_recent_activity_log_thinned_ind CREATE␠INDEX␠"mz_recent_activity_log_thinned_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s722␠AS␠"mz_internal"."mz_recent_activity_log_thinned"]␠("sql_hash") +mz_recent_sql_text_ind CREATE␠INDEX␠"mz_recent_sql_text_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s718␠AS␠"mz_internal"."mz_recent_sql_text"]␠("sql_hash") +mz_recent_storage_usage_ind CREATE␠INDEX␠"mz_recent_storage_usage_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s837␠AS␠"mz_catalog"."mz_recent_storage_usage"]␠("object_id") +mz_roles_ind CREATE␠INDEX␠"mz_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s509␠AS␠"mz_catalog"."mz_roles"]␠("id") mz_scheduling_elapsed_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_elapsed_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_elapsed_raw"␠("id",␠"worker_id") mz_scheduling_parks_histogram_raw_s2_primary_idx CREATE␠INDEX␠"mz_scheduling_parks_histogram_raw_s2_primary_idx"␠IN␠CLUSTER␠[s2]␠ON␠"mz_introspection"."mz_scheduling_parks_histogram_raw"␠("worker_id",␠"slept_for_ns",␠"requested_ns") -mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s487␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") -mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s522␠AS␠"mz_catalog"."mz_secrets"]␠("name") -mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s604␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") -mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s768␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") -mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_clusters"]␠("name") -mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s605␠AS␠"mz_internal"."mz_show_columns"]␠("id") -mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s614␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") -mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_databases"]␠("name") -mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s618␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") -mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s617␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") -mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s613␠AS␠"mz_internal"."mz_show_roles"]␠("name") -mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") -mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") -mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s616␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") -mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s615␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") -mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") -mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s612␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") -mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") -mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s733␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") -mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s705␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") -mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s706␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") -mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s499␠AS␠"mz_catalog"."mz_sinks"]␠("id") -mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s731␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") -mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s729␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") -mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s707␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") -mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s724␠AS␠"mz_internal"."mz_source_statuses"]␠("id") -mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s492␠AS␠"mz_catalog"."mz_sources"]␠("id") -mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s491␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") -mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s501␠AS␠"mz_catalog"."mz_types"]␠("schema_id") -mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s500␠AS␠"mz_catalog"."mz_views"]␠("schema_id") -mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s740␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") -mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s545␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") -pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s645␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") -pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s638␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") -pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s655␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") -pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s626␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") -pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s635␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") -pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s623␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") -pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s632␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") +mz_schemas_ind CREATE␠INDEX␠"mz_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s489␠AS␠"mz_catalog"."mz_schemas"]␠("database_id") +mz_secrets_ind CREATE␠INDEX␠"mz_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s524␠AS␠"mz_catalog"."mz_secrets"]␠("name") +mz_show_all_objects_ind CREATE␠INDEX␠"mz_show_all_objects_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s606␠AS␠"mz_internal"."mz_show_all_objects"]␠("schema_id") +mz_show_cluster_replicas_ind CREATE␠INDEX␠"mz_show_cluster_replicas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s770␠AS␠"mz_internal"."mz_show_cluster_replicas"]␠("cluster") +mz_show_clusters_ind CREATE␠INDEX␠"mz_show_clusters_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s608␠AS␠"mz_internal"."mz_show_clusters"]␠("name") +mz_show_columns_ind CREATE␠INDEX␠"mz_show_columns_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s607␠AS␠"mz_internal"."mz_show_columns"]␠("id") +mz_show_connections_ind CREATE␠INDEX␠"mz_show_connections_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s616␠AS␠"mz_internal"."mz_show_connections"]␠("schema_id") +mz_show_databases_ind CREATE␠INDEX␠"mz_show_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s610␠AS␠"mz_internal"."mz_show_databases"]␠("name") +mz_show_indexes_ind CREATE␠INDEX␠"mz_show_indexes_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s620␠AS␠"mz_internal"."mz_show_indexes"]␠("schema_id") +mz_show_materialized_views_ind CREATE␠INDEX␠"mz_show_materialized_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s619␠AS␠"mz_internal"."mz_show_materialized_views"]␠("schema_id") +mz_show_roles_ind CREATE␠INDEX␠"mz_show_roles_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s615␠AS␠"mz_internal"."mz_show_roles"]␠("name") +mz_show_schemas_ind CREATE␠INDEX␠"mz_show_schemas_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s611␠AS␠"mz_internal"."mz_show_schemas"]␠("database_id") +mz_show_secrets_ind CREATE␠INDEX␠"mz_show_secrets_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s609␠AS␠"mz_internal"."mz_show_secrets"]␠("schema_id") +mz_show_sinks_ind CREATE␠INDEX␠"mz_show_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s618␠AS␠"mz_internal"."mz_show_sinks"]␠("schema_id") +mz_show_sources_ind CREATE␠INDEX␠"mz_show_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s617␠AS␠"mz_internal"."mz_show_sources"]␠("schema_id") +mz_show_tables_ind CREATE␠INDEX␠"mz_show_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s612␠AS␠"mz_internal"."mz_show_tables"]␠("schema_id") +mz_show_types_ind CREATE␠INDEX␠"mz_show_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s614␠AS␠"mz_internal"."mz_show_types"]␠("schema_id") +mz_show_views_ind CREATE␠INDEX␠"mz_show_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s613␠AS␠"mz_internal"."mz_show_views"]␠("schema_id") +mz_sink_statistics_ind CREATE␠INDEX␠"mz_sink_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s735␠AS␠"mz_internal"."mz_sink_statistics"]␠("id",␠"replica_id") +mz_sink_status_history_ind CREATE␠INDEX␠"mz_sink_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s707␠AS␠"mz_internal"."mz_sink_status_history"]␠("sink_id") +mz_sink_statuses_ind CREATE␠INDEX␠"mz_sink_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s708␠AS␠"mz_internal"."mz_sink_statuses"]␠("id") +mz_sinks_ind CREATE␠INDEX␠"mz_sinks_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s501␠AS␠"mz_catalog"."mz_sinks"]␠("id") +mz_source_statistics_ind CREATE␠INDEX␠"mz_source_statistics_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s733␠AS␠"mz_internal"."mz_source_statistics"]␠("id",␠"replica_id") +mz_source_statistics_with_history_ind CREATE␠INDEX␠"mz_source_statistics_with_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s731␠AS␠"mz_internal"."mz_source_statistics_with_history"]␠("id",␠"replica_id") +mz_source_status_history_ind CREATE␠INDEX␠"mz_source_status_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s709␠AS␠"mz_internal"."mz_source_status_history"]␠("source_id") +mz_source_statuses_ind CREATE␠INDEX␠"mz_source_statuses_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s726␠AS␠"mz_internal"."mz_source_statuses"]␠("id") +mz_sources_ind CREATE␠INDEX␠"mz_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s494␠AS␠"mz_catalog"."mz_sources"]␠("id") +mz_tables_ind CREATE␠INDEX␠"mz_tables_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s493␠AS␠"mz_catalog"."mz_tables"]␠("schema_id") +mz_types_ind CREATE␠INDEX␠"mz_types_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s503␠AS␠"mz_catalog"."mz_types"]␠("schema_id") +mz_views_ind CREATE␠INDEX␠"mz_views_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s502␠AS␠"mz_catalog"."mz_views"]␠("schema_id") +mz_wallclock_global_lag_recent_history_ind CREATE␠INDEX␠"mz_wallclock_global_lag_recent_history_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s742␠AS␠"mz_internal"."mz_wallclock_global_lag_recent_history"]␠("object_id") +mz_webhook_sources_ind CREATE␠INDEX␠"mz_webhook_sources_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s547␠AS␠"mz_internal"."mz_webhook_sources"]␠("id") +pg_attrdef_all_databases_ind CREATE␠INDEX␠"pg_attrdef_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s647␠AS␠"mz_internal"."pg_attrdef_all_databases"]␠("oid",␠"adrelid",␠"adnum",␠"adbin",␠"adsrc") +pg_attribute_all_databases_ind CREATE␠INDEX␠"pg_attribute_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s640␠AS␠"mz_internal"."pg_attribute_all_databases"]␠("attrelid",␠"attname",␠"atttypid",␠"attlen",␠"attnum",␠"atttypmod",␠"attnotnull",␠"atthasdef",␠"attidentity",␠"attgenerated",␠"attisdropped",␠"attcollation",␠"database_name",␠"pg_type_database_name") +pg_authid_core_ind CREATE␠INDEX␠"pg_authid_core_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s657␠AS␠"mz_internal"."pg_authid_core"]␠("rolname") +pg_class_all_databases_ind CREATE␠INDEX␠"pg_class_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s628␠AS␠"mz_internal"."pg_class_all_databases"]␠("relname") +pg_description_all_databases_ind CREATE␠INDEX␠"pg_description_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s637␠AS␠"mz_internal"."pg_description_all_databases"]␠("objoid",␠"classoid",␠"objsubid",␠"description",␠"oid_database_name",␠"class_database_name") +pg_namespace_all_databases_ind CREATE␠INDEX␠"pg_namespace_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s625␠AS␠"mz_internal"."pg_namespace_all_databases"]␠("nspname") +pg_type_all_databases_ind CREATE␠INDEX␠"pg_type_all_databases_ind"␠IN␠CLUSTER␠[s2]␠ON␠[s634␠AS␠"mz_internal"."pg_type_all_databases"]␠("oid") # Record all transitive dependencies (tables, sources, views, mvs) of indexes on # the mz_catalog_server cluster. @@ -558,6 +558,8 @@ mz_object_arrangement_sizes replica_id mz_object_arrangement_sizes size mz_object_dependencies object_id mz_object_dependencies referenced_object_id +mz_object_dependencies_raw object_id +mz_object_dependencies_raw referenced_object_id mz_object_graph_edges dependency_id mz_object_graph_edges object_id mz_object_history cluster_id diff --git a/test/sqllogictest/mz_object_dependencies.slt b/test/sqllogictest/mz_object_dependencies.slt new file mode 100644 index 0000000000000..fe7463f151f90 --- /dev/null +++ b/test/sqllogictest/mz_object_dependencies.slt @@ -0,0 +1,313 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Behavioural tests for mz_internal.mz_object_dependencies. + +mode cockroach + +reset-server + +# All user objects live in a scratch schema owned by the test role, so the +# cascade at the end of the file can clean up everything in one statement. +# The public schema is owned by mz_system, so it cannot play that role. +statement ok +CREATE SCHEMA deps + +statement ok +SET search_path = deps + +statement ok +CREATE TABLE t (a int, b text NOT NULL) + +statement ok +CREATE VIEW v AS SELECT pg_catalog.abs(a), b::text FROM t WHERE a > 42 + +# --- relation and function edges from user items ------------------------------ + +# v depends on t (relation reference, stored as an id in create_sql). +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'v') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +# v depends on pg_catalog.abs. Function references print as plain names in +# stored SQL and are recovered by name from the GidMapping rows. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'v') + AND referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'abs') +---- +1 + +# v depends on the type named in its definition (text from the cast), which +# prints as an id reference in stored SQL. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_views WHERE name = 'v') +---- +text + +# t depends on its column types. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +int4 +text + +# --- array types --------------------------------------------------------------- + +statement ok +CREATE TABLE arr (a int4[]) + +# int4[] yields an edge to the array type _int4 and none to the element type +# int4, the same set a direct reference to _int4 produces. Name resolution +# treats both spellings as referencing the array type alone. +query T rowsort +SELECT ty.name +FROM mz_internal.mz_object_dependencies dep +JOIN mz_types ty ON ty.id = dep.referenced_object_id +WHERE dep.object_id = (SELECT id FROM mz_tables WHERE name = 'arr') +---- +_int4 + +# --- connection and secret edges ------------------------------------------------ + +statement ok +CREATE SECRET pw AS 'hunter2' + +statement ok +CREATE CONNECTION kc TO KAFKA (BROKER 'kafka:9092', SASL MECHANISMS 'PLAIN', SASL USERNAME 'u', SASL PASSWORD = SECRET pw) WITH (VALIDATE = false) + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_connections WHERE name = 'kc') + AND referenced_object_id = (SELECT id FROM mz_secrets WHERE name = 'pw') +---- +1 + +# --- temporary items are excluded ---------------------------------------------- + +statement ok +CREATE TEMPORARY VIEW tv AS SELECT * FROM t + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'tv') +---- +0 + +# --- materialized view edges ----------------------------------------------------- + +statement ok +CREATE MATERIALIZED VIEW umv AS SELECT a + 1 AS a1 FROM t + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_materialized_views WHERE name = 'umv') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +# --- index edges ------------------------------------------------------------------- + +statement ok +CREATE INDEX fidx ON t (abs(a)) + +# The index depends on the relation it indexes and on the function its key +# calls. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_indexes WHERE name = 'fidx') + AND referenced_object_id = (SELECT id FROM mz_tables WHERE name = 't') +---- +1 + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_indexes WHERE name = 'fidx') + AND referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'abs') +---- +1 + +# --- custom types ------------------------------------------------------------------ + +statement ok +CREATE TYPE custom_list AS LIST (ELEMENT TYPE = int4) + +# A custom type depends on its element type: a type -> type edge. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_types WHERE name = 'custom_list') + AND referenced_object_id = (SELECT id FROM mz_types WHERE name = 'int4') +---- +1 + +# --- tables from sources ------------------------------------------------------------- + +statement ok +CREATE SOURCE lg FROM LOAD GENERATOR COUNTER + +statement ok +CREATE TABLE lg_tbl FROM SOURCE lg (REFERENCE counter) + +# The table's create_sql references its source by id outside any query. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_tables WHERE name = 'lg_tbl') + AND referenced_object_id = (SELECT id FROM mz_sources WHERE name = 'lg') +---- +1 + +# --- builtin edges --------------------------------------------------------------- + +# mz_catalog.mz_indexes is itself a materialized view over mz_catalog_raw. +# Its edge comes from the VALUES inlined at generation time, joined to +# GidMapping rows at query time. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_catalog' AND o.name = 'mz_indexes' + AND rs.name = 'mz_internal' AND r.name = 'mz_catalog_raw' +---- +1 + +# Builtin views also depend on the functions they call, recorded by name at +# generation time. mz_catalog.mz_indexes calls mz_internal.parse_catalog_id. +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +WHERE os.name = 'mz_catalog' AND o.name = 'mz_indexes' + AND dep.referenced_object_id IN (SELECT id FROM mz_functions WHERE name = 'parse_catalog_id') +---- +true + +# Introspection source indexes depend on their log. +query T +SELECT count(*) > 0 FROM mz_internal.mz_object_dependencies WHERE object_id LIKE 'si%' +---- +true + +# The relation itself must be reported as a materialized view. +query T +SELECT count(*) FROM mz_materialized_views WHERE name = 'mz_object_dependencies' +---- +1 + +query T +SELECT count(*) FROM mz_internal.mz_ontology_entity_types WHERE name = 'object_dependency' +---- +1 + +# The generating view carries its own outgoing edges. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies_raw' + AND rs.name = 'mz_internal' AND r.name = 'mz_catalog_raw' +---- +1 + +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies_raw' + AND dep.referenced_object_id IN ( + SELECT id FROM mz_functions WHERE name = 'parse_catalog_item_references' + ) +---- +true + +# mz_object_dependencies reads mz_object_dependencies_raw, so the edge between +# the two is recorded like any other. +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies dep +JOIN mz_objects o ON o.id = dep.object_id +JOIN mz_schemas os ON os.id = o.schema_id +JOIN mz_objects r ON r.id = dep.referenced_object_id +JOIN mz_schemas rs ON rs.id = r.schema_id +WHERE os.name = 'mz_internal' AND o.name = 'mz_object_dependencies' + AND rs.name = 'mz_internal' AND r.name = 'mz_object_dependencies_raw' +---- +1 + +# --- dropped items leave no edges -------------------------------------------------- + +statement ok +CREATE VIEW dropme AS SELECT a FROM t + +query T +SELECT count(*) > 0 +FROM mz_internal.mz_object_dependencies +WHERE object_id = (SELECT id FROM mz_views WHERE name = 'dropme') +---- +true + +statement ok +DROP VIEW dropme + +# --- structural invariants ------------------------------------------------------- + +# No duplicate edges: ResolvedIds was a map, so the packed table never +# produced duplicates, and consumers (e.g. transitive closures) assume this. +query T +SELECT count(*) FROM ( + SELECT object_id, referenced_object_id + FROM mz_internal.mz_object_dependencies + GROUP BY object_id, referenced_object_id + HAVING count(*) > 1 +) +---- +0 + +# No self-edges. +query T +SELECT count(*) FROM mz_internal.mz_object_dependencies WHERE object_id = referenced_object_id +---- +0 + +# No edge outlives its object: cascading away the whole scratch schema drops +# every user object created above, so any user edge still present is a leak. +statement ok +DROP SCHEMA deps CASCADE + +query T +SELECT count(*) +FROM mz_internal.mz_object_dependencies +WHERE object_id LIKE 'u%' OR referenced_object_id LIKE 'u%' +---- +0 diff --git a/test/sqllogictest/oid.slt b/test/sqllogictest/oid.slt index fe4fdaaebd174..8bea7512a2c74 100644 --- a/test/sqllogictest/oid.slt +++ b/test/sqllogictest/oid.slt @@ -1254,3 +1254,5 @@ SELECT oid, name FROM mz_objects WHERE id LIKE 's%' AND oid < 20000 ORDER BY oid 17122 mz_object_hydration_history 17123 mz_cluster_replica_resource_usage 17124 mz_replica_hydration_history +17125 parse_catalog_item_references +17126 mz_object_dependencies_raw diff --git a/test/testdrive/catalog.td b/test/testdrive/catalog.td index 12074488a6a79..645afa00580b8 100644 --- a/test/testdrive/catalog.td +++ b/test/testdrive/catalog.td @@ -608,7 +608,6 @@ mz_history_retention_strategies "" mz_license_keys "" mz_materialized_view_refresh_strategies "" mz_object_arrangement_size_history "" -mz_object_dependencies "" mz_object_global_ids "" mz_object_hydration_history "" mz_optimizer_notices "" @@ -649,6 +648,7 @@ mz_mcp_data_product_details "" mz_mcp_data_products "" mz_notices "" mz_notices_redacted "" +mz_object_dependencies_raw "" mz_object_fully_qualified_names "" mz_object_graph_edges "" mz_object_history "" @@ -734,6 +734,7 @@ mz_metric_sinks mz_catalog_server "" mz_mysql_source_tables mz_catalog_server "" mz_network_policies mz_catalog_server "" mz_network_policy_rules mz_catalog_server "" +mz_object_dependencies mz_catalog_server "" mz_overridden_system_parameters mz_catalog_server "" mz_pending_cluster_replicas mz_catalog_server "" mz_postgres_source_tables mz_catalog_server "" @@ -837,7 +838,7 @@ test_table "" # `SHOW TABLES` and `mz_tables` should agree. > SELECT COUNT(*) FROM mz_tables WHERE id LIKE 's%' -31 +30 # There is one entry in mz_indexes for each field_number/expression of the index. > SELECT COUNT(id) FROM mz_indexes WHERE id LIKE 's%' diff --git a/test/testdrive/mz-depends.td b/test/testdrive/mz-depends.td index 4f3b29b2ab948..6ba32dfec7a6e 100644 --- a/test/testdrive/mz-depends.td +++ b/test/testdrive/mz-depends.td @@ -65,6 +65,36 @@ conn dep_conn pg_conn ssh_conn +# A function referenced both by id and by name in one statement yields a +# single edge. + +$ set-from-sql var=gs-id +SELECT DISTINCT id FROM mz_functions +WHERE name = 'generate_series' + AND schema_id IN (SELECT id FROM mz_schemas WHERE name = 'pg_catalog') + +> CREATE VIEW dup_func_ref AS + SELECT by_id.g, by_name.h + FROM [${gs-id} AS pg_catalog.generate_series](1, 3) AS by_id(g), + generate_series(4, 6) AS by_name(h); + +# Both spellings persist and land in different reference buckets. +> SELECT refs->'ids' ? '${gs-id}' AS by_id, + refs->'named_funcs' @> '[{"schema": "pg_catalog", "name": "generate_series"}]' AS by_name + FROM ( + SELECT mz_internal.parse_catalog_item_references(create_sql) AS refs + FROM mz_views WHERE name = 'dup_func_ref' + ); +by_id by_name +------------- +true true + +# ... yet exactly one edge to the function. +> SELECT count(*) FROM mz_internal.mz_object_dependencies + WHERE object_id = (SELECT id FROM mz_views WHERE name = 'dup_func_ref') + AND referenced_object_id = '${gs-id}'; +1 + # Assert that we actually are populating all the system objects. > SELECT COUNT(*) > 200 FROM mz_internal.mz_object_dependencies WHERE object_id LIKE 's%' true