docs: document the tiered data and Iceberg format version sync properties - #472
Closed
alejandrodnm wants to merge 3 commits into
Closed
alejandrodnm wants to merge 3 commits into
alejandrodnm wants to merge 3 commits into
Conversation
Tiered data was listed as a flat limitation of TigerLake sync. It is now opt-in per table, so document the tigerlake.enable_tiered_reads property, add a Tiered data section covering the set-at-sync-start restriction and its independence from the timescaledb.enable_tiered_reads GUC, and make the limitation conditional instead of absolute.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
tigerlake.iceberg_spec_version shipped undocumented. Add it to the reference statement and option list, and add an Iceberg format version section covering the accepted values, the v3 default for new syncs, the set-at-sync-start restriction, and the fact that pre-existing syncs stay on v2.
Collaborator
|
hey @alejandrodnm closing this as it appears to be resolved with the linked PR. |
Contributor
Author
|
@erinmikailstaples the linked PR is to expose the feature in the UI. That's behind a feature flag, we haven't deployed this features yet, the docs are still needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Two
tigerlake.*table properties shipped intimescaledb-lake0.7.2 without docs. Both are set on theALTER TABLEstatement that starts an Iceberg sync, so both belong on the same page.Tiered data. Syncing tiered data used to be impossible, and this page listed it as a flat limitation. It is now opt-in per table via
tigerlake.enable_tiered_reads, so the limitation is no longer absolute. Adds a Tiered data subsection covering how to turn it on, the fact that it can only be set in the statement that starts the sync, its independence from thetimescaledb.enable_tiered_readsGUC, and the snapshot cost of including tiered history. The limitation bullet becomes conditional rather than absolute.Iceberg format version.
tigerlake.iceberg_spec_versionwas never documented at all; the docs had no mention of Iceberg format versions anywhere. Adds an Iceberg format version subsection covering the accepted values, the v3 default for new syncs, the set-at-sync-start restriction, and the fact that tables that started syncing before the property existed stay on v2.Both properties are also added to the reference
ALTER TABLEstatement and its option list.Every behavioral claim was verified against the extension source (
timescaledb-lake) rather than inferred: the accepted values, the set-at-sync-start restriction, and the errors on changing either property afterwards come from the validation insrc/sync_config.rs; the defaults come fromsql/tables.sql; and the "pre-existing syncs stay on v2" behavior comes from the upgrade script insql/latest-dev.sql.Affected pages
/integrate/connectors/destination/tigerlake
🧪 Component kitchen sink — check for visual regressions if you changed components, styles, or dependencies.
Related Issues
Issue: #number
Extension side: timescale/timescaledb-lake#142 added
iceberg_spec_version, timescale/timescaledb-lake#148 addedenable_tiered_reads, and timescale/timescaledb-lake#151 fixes the latter's upgrade path.Checklist before requesting a review
Notes on the unchecked items:
Neither feature is public yet. Both landed after the
v0.7.1tag and sit in 0.7.2-dev, unreleased. Please schedule this for after that release. Draft until then.One documented behavior is not live yet. The Tiered data subsection states that setting
tigerlake.enable_tiered_readstofalsekeeps tiered data out of Iceberg even when thetimescaledb.enable_tiered_readsGUC is enabled for the database. That is the agreed target behavior, but todayfalseis a no-op that inherits the GUC. It depends on timescale/timescaledb-lake#151 plus a matching sink change landing first. Worth re-reading that bullet before merge. Nothing in the Iceberg format version subsection has this caveat.No images, videos, or component changes, so those items are not applicable. I verified rendering locally and confirmed both new anchors and every internal and external link resolve, but I did not run an end-to-end sync of either kind, hence the untested-code item.