feat: narrow stream delete formats by destination catalog by vikaxsh · Pull Request #1290 · datazip-inc/olake

vikaxsh · 2026-10-06T12:46:28Z

Description

Discover now intersects each stream's available_update_types with what the destination's
Iceberg catalog can apply, on top of the target query engines.

--destination on discover is parsed (not connected to) for the raw catalog_type,
read before iceberg's validation folds REST-family catalogs into rest.
New per-catalog write matrix, sourced from vendor docs (Oct 2026):
- nessie: no dv (table spec v3 unsupported)
- unity: dv only (no identifier fields; Databricks supports no eq/pos delete files)
- horizon: no eq (rejected from external engines)

No --destination keeps today's behavior: delete formats are constrained by engines only.

Fixes # (issue)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Scenario A
Scenario B

Screenshots or Recordings

Documentation

Documentation Link: [link to README, olake.io/docs, or olake-docs]
N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

github-actions · 2026-10-06T12:59:19Z

🔍 OpenCodeReview found 1 issue(s) in this PR.

✅ Successfully posted inline: 1 comment(s)

…datazip-inc/olake into feat/catalog-aware-update-types

…eat/catalog-aware-update-types

hash-data

See if we can make it easy implementation seems complex

hash-data · 2026-10-26T09:16:48Z

+	if destinationConfigPath != "not-set" && destinationConfigPath != "" {
+		destinationConfig = &types.WriterConfig{}
+		if err := utils.UnmarshalFile(destinationConfigPath, destinationConfig, true); err != nil {
+			return err
+		}
+	}
+	catalogType, err := types.CatalogTypeFromConfig(destinationConfig)
+	if err != nil {
+		return errs.Precondition(errs.ConfigInvalid, codeDestinationConfigInvalid, err)
+	}


what if older ui and new driver version being used?, like old people try to create new job?

for them without destination flag update type would be based on query engines only

hash-data · 2026-10-26T09:27:52Z

+
+	catalog := writer.CatalogType
+	if catalog == "" {
+		catalog = defaultCatalogType


so for parquet writer is it going to be default ?

function returns early for parquet dest

hash-data · 2026-10-26T09:37:26Z

+	if destinationConfigPath != "not-set" && destinationConfigPath != "" {
+		destinationConfig = &types.WriterConfig{}
+		if err := utils.UnmarshalFile(destinationConfigPath, destinationConfig, true); err != nil {
+			return err
+		}
+	}
+	catalogType, err := types.CatalogTypeFromConfig(destinationConfig)
+	if err != nil {
+		return errs.Precondition(errs.ConfigInvalid, codeDestinationConfigInvalid, err)
+	}
+
+	constraints := types.UpdateTypeConstraints{
+		Engines:    engines,
+		Catalog:    catalogType,
+		AppendOnly: types.DestinationIsAppendOnly(destinationConfig),
+	}
+	// Only upsert needs a delete format and append writes none, so the job stays valid: each
+	// stream gets an empty available_update_types for the UI to report. Expected, so silent,
+	// for an append-only destination.
+	if !constraints.AppendOnly && len(constraints.Available()) == 0 {


let us simplify logic, seems unenecessary complex

There isn’t much scope for simplification since this function handles three different responsibilities. We could split it into smaller helper functions, but it wouldn’t provide much benefit

hash-data · 2026-10-26T09:46:22Z

 		return fmt.Errorf("update mode %q is not readable by the target query engines; available are %v", m, available)
 	}


we need to update this error message

hash-data · 2026-10-26T09:48:52Z

+var icebergCatalogs = []CatalogSpec{
+	{Catalog: "glue", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	{Catalog: "jdbc", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	{Catalog: "hive", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	{Catalog: "rest", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	{Catalog: "lakekeeper", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	// Nessie does not support Iceberg table spec v3, so no deletion vectors.
+	{Catalog: "nessie", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition}},
+	{Catalog: "s3tables", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	// Unity rejects identifier columns, which eq keys on (without them OLake falls back to
+	// append), and Databricks supports neither eq nor pos delete files, only deletion vectors.
+	{Catalog: "unity", Supports: []UpdateType{UpdateTypeDeletionVector}},
+	{Catalog: "polaris", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	// Deletion vectors are in Preview on the BigLake (Lakehouse runtime) catalog.
+	{Catalog: "biglake", Supports: []UpdateType{UpdateTypeEquality, UpdateTypePosition, UpdateTypeDeletionVector}},
+	// Snowflake rejects equality deletes from external engines. dv works only on tables created
+	// at v3: Horizon forbids upgrading an existing v2 table, which switching a stream to dv needs.


let us only include catalog which does not support all delete type, will be easy

…eat/catalog-aware-update-types

…datazip-inc/olake into feat/catalog-aware-update-types

feat: narrow stream delete formats by destination catalog

6872946

vikaxsh had a problem deploying to ai-code-review October 6, 2026 12:46 — with GitHub Actions Error

Merge branch 'staging' into feat/catalog-aware-update-types

f7e08dd

vikaxsh deployed to ai-code-review October 6, 2026 12:46 — with GitHub Actions Active

vikaxsh deployed to integration_tests October 6, 2026 12:46 — with GitHub Actions Active

github-actions Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread protocol/discover.go Outdated

Comment thread protocol/discover.go Outdated

vikaxsh added 2 commits October 6, 2026 18:50

feat: No qualifying format no longer fails discover

53f5068

Merge branch 'feat/catalog-aware-update-types' of https://github.com/…

3509b3c

…datazip-inc/olake into feat/catalog-aware-update-types

vikaxsh had a problem deploying to ai-code-review October 6, 2026 13:21 — with GitHub Actions Failure

vikaxsh requested a deployment to integration_tests October 6, 2026 13:21 — with GitHub Actions Waiting

vikaxsh deployed to ai-code-review October 6, 2026 13:24 — with GitHub Actions Active

vikaxsh added 2 commits October 16, 2026 18:35

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

7367ee1

…eat/catalog-aware-update-types

feat: add append mode true for s3 dest

618d0d9

vikaxsh deployed to integration_tests October 16, 2026 13:20 — with GitHub Actions Active

deepanshupal09-datazip mentioned this pull request Oct 21, 2026

fix: interoperability ui fixes datazip-inc/olake-ui#462

Open

1 of 6 tasks

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

a1cf45b

…eat/catalog-aware-update-types

vikaxsh deployed to ai-code-review October 26, 2026 06:34 — with GitHub Actions Active

vikaxsh deployed to integration_tests October 26, 2026 06:34 — with GitHub Actions Active

github-actions Bot reviewed Oct 26, 2026

View reviewed changes

Comment thread types/delete_mode.go Outdated

hash-data reviewed Oct 26, 2026

View reviewed changes

Merge branch 'staging' into feat/catalog-aware-update-types

8115d10

vikaxsh had a problem deploying to ai-code-review October 27, 2026 05:13 — with GitHub Actions Error

vikaxsh deployed to integration_tests October 27, 2026 05:14 — with GitHub Actions Active

vikaxsh added 2 commits October 27, 2026 10:51

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

da75427

…eat/catalog-aware-update-types

Merge branch 'feat/catalog-aware-update-types' of https://github.com/…

de7daa5

…datazip-inc/olake into feat/catalog-aware-update-types

vikaxsh requested a deployment to integration_tests October 27, 2026 05:23 — with GitHub Actions Waiting

vikaxsh had a problem deploying to ai-code-review October 27, 2026 05:23 — with GitHub Actions Error

chore: refactor code

ba2ed77

vikaxsh had a problem deploying to ai-code-review October 27, 2026 05:30 — with GitHub Actions Error

vikaxsh deployed to integration_tests October 27, 2026 05:30 — with GitHub Actions Active

chore: resolve review comments

b40009b

vikaxsh deployed to integration_tests October 27, 2026 06:46 — with GitHub Actions Active

vikaxsh had a problem deploying to ai-code-review October 27, 2026 06:47 — with GitHub Actions Error

chore: simplified logic

088d646

vikaxsh requested a deployment to ai-code-review October 27, 2026 07:06 — with GitHub Actions Waiting

vikaxsh requested a deployment to integration_tests October 27, 2026 07:06 — with GitHub Actions Waiting

		return fmt.Errorf("update mode %q is not readable by the target query engines; available are %v", m, available)
		}

Conversation

vikaxsh commented Oct 6, 2026

Description

Type of change

How Has This Been Tested?

Screenshots or Recordings

Documentation

Related PR's (If Any):

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hash-data left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch is waiting to be deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 6, 2026 •

edited

Loading