From 3f2cdddbcd4ebe7c81e2544b0ae41f9ac121907d Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:04:04 -0700 Subject: [PATCH 1/6] feat(format): define provider-managed clustering metadata --- docs/src/format/table/.pages | 1 + docs/src/format/table/clustering.md | 114 ++++++++++++++++++++ docs/src/format/table/versioning.md | 4 +- protos/clustering.proto | 34 ++++++ protos/fragment_metadata.proto | 8 ++ protos/table.proto | 13 +++ rust/lance-table/benches/manifest_intern.rs | 2 + rust/lance-table/src/feature_flags.rs | 22 ++++ rust/lance-table/src/format/fragment.rs | 1 + rust/lance-table/src/format/index.rs | 1 + rust/lance-table/src/format/manifest.rs | 75 +++++++++++++ 11 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 docs/src/format/table/clustering.md create mode 100644 protos/clustering.proto diff --git a/docs/src/format/table/.pages b/docs/src/format/table/.pages index 064c080e5f4..e1fe5b007c3 100644 --- a/docs/src/format/table/.pages +++ b/docs/src/format/table/.pages @@ -7,5 +7,6 @@ nav: - Branch & Tag: branch_tag.md - Row ID & Lineage: row_id_lineage.md - Data Overlay Files: data_overlay_file.md + - Clustering Metadata: clustering.md - Fragment Metadata Tree: fragment_metadata.md - MemTable & WAL: mem_wal.md diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md new file mode 100644 index 00000000000..3f3988f32ac --- /dev/null +++ b/docs/src/format/table/clustering.md @@ -0,0 +1,114 @@ +# Clustering Metadata + +Clustering organizes data to improve query pruning. Like Delta, Lance separates +the table's current policy from information about individual outputs. Providers +choose the layout and maintenance logic. This proposal defines the format in +`clustering.proto`; the writer feature remains unsupported until implemented. + +## Stored Information + +`Manifest.clustering` holds the current `Clustering` configuration: + +- `columns`: an ordered, non-empty list of distinct, non-negative schema field + IDs for existing clustering columns. +- `provider`: a non-empty, case-sensitive implementation name. +- `version`: a non-zero, provider-assigned layout/configuration identifier. +- `provider_metadata`: `map` containing provider configuration. + +`DataFragment.clustering` and `IndexMetadata.clustering` each hold optional +`ClusteringInfo`: `provider`, `version`, and `metadata: map`. +Providers define keys and string encodings, such as bucket, range, or group +identifiers. Empty maps are valid. A segment's information describes its own +grouping; it does not relabel its input fragments. + +`version` is not a dataset version or software release. Within a table, including +branches, `(provider, version)` must keep the same layout meaning. Layout changes +use a new version; providers manage any history they need. An unknown old version +must not be interpreted using a different current configuration. + +## Main Rules + +- Enabling clustering does not require every append to cluster its output or + impose a deadline for organizing existing data. +- Changing or removing configuration does not rewrite history or erase existing + fragment and segment information. Historical reclustering is a separate operation. +- Providers may take over other providers' fragments, recluster them, and write + their own information on the outputs. Lance imposes no provider ownership boundary. +- When configuration is present, `columns` defines the clustering keys. Writers + clear the older `Field.unenforced_clustering_key_position` and + `Field.unenforced_clustering_key` hints to avoid conflicting definitions. + +## Clustering, Compaction, and Indexing + +Providers choose the fragments to process and return groups of fragments: + +- **Clustering:** the provider chooses which fragments to recluster together, + reorganizes their rows, and writes its information on the new fragments. +- **Compaction:** the provider compacts each group and writes clustering + information for the new fragments. +- **Indexing:** each index has its own groups. The provider builds each segment + and writes its clustering information; different index types may use different + grouping sizes over the same fragment layout. + +Providers may reuse Lance's builders. Lance validates inputs against concurrent +changes and commits the outputs and metadata atomically. Runtime interface +signatures are outside this format contract. + +Without an available provider compaction strategy, default compaction skips +clustered fragments and processes unclustered ones normally. An explicit ordinary +compaction may override this; its output is unmarked. Provider failures are +reported rather than silently switching strategies. Without a provider indexing +strategy, normal index building remains available, producing unmarked segments. +Fragments without clustering information use Lance's default compaction and +indexing unless a provider selects them for its own operations. + +## Ordinary Maintenance + +- Unrelated commits preserve configuration and unchanged fragments' and + segments' information, including unknown metadata keys and values. +- Ordinary data rewrites do not inherit fragment clustering information. + Column replacement and overlays clear it even when the fragment ID remains + unchanged or only non-clustering columns change. No layout-impact analysis is needed. +- Deletion-vector-only changes preserve the original fragment's information. + Providers must tolerate deleted rows and revalidate maintenance bookkeeping. +- Ordinary index merge, rebuild, and remap do not inherit information on the + rewritten segments. Data fragments' clustering information is unaffected. +- Provider operations set their outputs' information. Metadata maps are not + automatically copied or merged from inputs. + +Dropping a clustering field or changing its type removes the current configuration. +Renaming while retaining field IDs and types preserves it. Overwrite may retain +configuration only when its field IDs and types remain compatible. Clearing output +information does not otherwise disable the policy. Restore restores the selected +snapshot's information; clone or import preserves layout identity or clears +conflicting claims. Metadata paths acquire no file-retention or clone semantics. + +## Statistics and Reads + +Clustering writes collect zone-level write seeds for clustering columns using +Lance's defined Arrow Rust comparison semantics. Consumers may use zones directly +or aggregate them into fragment or segment statistics; no particular statistics +index is required. Missing or unusable statistics keep the data as a query candidate. + +Readers may ignore clustering information. Provider metadata alone cannot prove +row order or justify pruning. Statistics, index coverage, FRI translation, and +deletions retain their own contracts. Metadata cannot impose hidden requirements +for correct queries or ordinary writes. + +## Compatibility + +A snapshot referencing table configuration, any fragment clustering information, +or any index segment clustering information must set `FLAG_CLUSTERING_METADATA` +(`1 << 17`) in `writer_feature_flags`, not `reader_feature_flags`. This includes +fragments in a fragment metadata tree and indices in the index section. + +Older writers must refuse to write; readers need no clustering capability. +Supporting writers preserve or clear information using the rules above without +understanding every provider. Clear the flag only when all three kinds of +information are absent; disabling table clustering alone is insufficient. +Providers share one flag, independent of layout version. FRI and write seeds +keep their own feature requirements. This reservation does not enable clustering +APIs or metadata-aware writes. + +Sources: [Delta clustered-table protocol](https://github.com/delta-io/delta/blob/ef3e91b509365cdc2b23c1afdbbab3b4d3828d63/PROTOCOL.md#clustered-table) +and [changing clustering columns](https://docs.delta.io/delta-clustering/#change-clustering-columns). diff --git a/docs/src/format/table/versioning.md b/docs/src/format/table/versioning.md index e4d77b9f273..c38cdae5c75 100644 --- a/docs/src/format/table/versioning.md +++ b/docs/src/format/table/versioning.md @@ -40,8 +40,10 @@ they should return an "unsupported" error on any read or write operation. | `2^14` | `FLAG_MANAGED_BLOBS` | Yes | Yes | Blob descriptors independently address owned objects. Readers resolve the descriptor's base and path; cleanup preserves objects referenced by protected snapshots, independently of the original data file. Both bits remain set on later versions, including restores. | | `2^15` | `FLAG_NON_REUSABLE_FIELD_IDS` | No | Yes | The manifest sets `max_allocated_field_id`, and a writer must assign new field IDs above it. Before activation, all clients that can write to the dataset must enforce writer feature flags. See [Field IDs](schema.md#field-ids). | +| `2^17` | `FLAG_CLUSTERING_METADATA` | No | Yes | The snapshot references [clustering configuration or fragment/index segment information](clustering.md). Reserved, unsupported until writers implement its maintenance rules. Readers may ignore clustering metadata. | + -Bits at positions 16 and above (`2^16`, `2^17`, and so on) are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version and Managed Blob reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. +Bit 17 is reserved for clustering metadata but remains unsupported. Other bits at positions 16 and above are unknown. Unknown or unsupported flags cause implementations to reject the corresponding read or write operation with an "unsupported" error. The paired mixed-version and Managed Blob reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. Publishing a new data file containing a Blob v2 field activates the Managed Blob capability, including files containing only inline values. Metadata-only updates and deletion vectors on existing files do not activate it. Once activated, removing Blob fields or restoring an older snapshot does not clear it. Before activation, all maintenance clients must support Managed Blobs: the flag cannot revoke old handles or prevent older clients from running cleanup through an unflagged historical snapshot. See [client compatibility](../../guide/blob.md#managed-objects-and-client-compatibility). diff --git a/protos/clustering.proto b/protos/clustering.proto new file mode 100644 index 00000000000..0c9d97ba62e --- /dev/null +++ b/protos/clustering.proto @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +syntax = "proto3"; + +package lance.table; + +// Current table-level clustering configuration. Providers manage layout history. +message Clustering { + // Ordered, non-empty list of distinct, existing schema field IDs. + repeated int32 columns = 1; + // Non-empty, case-sensitive clustering implementation name. + string provider = 2; + // Non-zero layout/configuration identifier, not a dataset or software version. + uint64 version = 3; + /* Provider-defined configuration. Empty is valid. Providers define keys and + * value encodings; unrelated writes preserve unknown keys and values. + */ + map provider_metadata = 4; +} + +// Clustering information shared by fragments and index segments. +message ClusteringInfo { + // Non-empty, case-sensitive implementation name that organized this output. + string provider = 1; + // Non-zero identifier of the provider's layout/configuration for this output. + uint64 version = 2; + /* Provider-defined information, such as group, bucket, or range identifiers. + * Empty is valid. It is not proof of index coverage, statistics, or row order. + * Ordinary rewrites do not inherit this information; provider operations set + * it for their outputs. Unchanged objects preserve unknown keys and values. + */ + map metadata = 3; +} diff --git a/protos/fragment_metadata.proto b/protos/fragment_metadata.proto index 63e2651d1e8..861bb103018 100644 --- a/protos/fragment_metadata.proto +++ b/protos/fragment_metadata.proto @@ -5,6 +5,8 @@ syntax = "proto3"; package lance.table; +import "clustering.proto"; + /* A DataFragment is a set of files which represent the different columns of the same * rows. If column exists in the schema of a dataset, but the file for that column does * not exist within a DataFragment of that dataset, that column consists entirely of @@ -85,6 +87,12 @@ message DataFragment { * `deletion_file.num_deleted_rows` from this value. */ uint64 physical_rows = 4; + + /* Optional provider, layout version, and metadata for this fragment. + * Ordinary rewrites do not inherit it; deletion-vector-only changes preserve + * it. Absence makes no clustering claim. Requires writer flag 1 << 17. + */ + ClusteringInfo clustering = 15; } message DataFile { diff --git a/protos/table.proto b/protos/table.proto index 2370324a346..5df8b053604 100644 --- a/protos/table.proto +++ b/protos/table.proto @@ -8,6 +8,7 @@ package lance.table; import "google/protobuf/any.proto"; import "google/protobuf/timestamp.proto"; import "file.proto"; +import "clustering.proto"; import public "fragment_metadata.proto"; /* @@ -290,6 +291,12 @@ message Manifest { * both be set or both be absent. A mismatch is invalid. */ optional int32 max_allocated_field_id = 24; + + /* Optional current clustering policy. Absence disables new clustering work + * without clearing fragment or index segment information. Requires the + * clustering writer feature flag (1 << 17); no clustering reader flag. + */ + Clustering clustering = 25; } // Manifest // external dataset base path @@ -415,6 +422,12 @@ message IndexMetadata { * withdraw the payload while leaving this declaration in place. */ repeated int32 covering_fields = 11; + + /* Optional clustering information for this segment's grouping. Ordinary + * merge, rebuild, and remap do not inherit it; provider operations set it for + * their outputs. Absence makes no clustering claim. Requires writer flag 1 << 17. + */ + ClusteringInfo clustering = 12; } // Metadata about a single file within an index segment. diff --git a/rust/lance-table/benches/manifest_intern.rs b/rust/lance-table/benches/manifest_intern.rs index 81bd57c1a22..358b42a6a51 100644 --- a/rust/lance-table/benches/manifest_intern.rs +++ b/rust/lance-table/benches/manifest_intern.rs @@ -63,6 +63,7 @@ fn make_uniform_pb_fragments(n: u64, num_fields: usize) -> Vec deletion_file: None, row_id_sequence: None, physical_rows: 1000, + clustering: None, last_updated_at_version_sequence: Some( pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions( version_bytes.clone(), @@ -140,6 +141,7 @@ fn make_diverse_pb_fragments( deletion_file: None, row_id_sequence: None, physical_rows: 1000, + clustering: None, last_updated_at_version_sequence: Some( pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions( version_payloads[version_idx].clone(), diff --git a/rust/lance-table/src/feature_flags.rs b/rust/lance-table/src/feature_flags.rs index 435a2a7fe21..2f2390a71c7 100644 --- a/rust/lance-table/src/feature_flags.rs +++ b/rust/lance-table/src/feature_flags.rs @@ -109,6 +109,10 @@ pub const MANAGED_BLOBS_META_KEY: &str = "lance.blob.managed"; /// Writers must understand this allocation contract. It does not change how /// readers interpret the schema or data files. pub const FLAG_NON_REUSABLE_FIELD_IDS: u64 = 1 << 15; +/// Writer-only protection for table, fragment, and index segment clustering metadata. +/// Reserved beyond the supported range until writers implement its maintenance rules. +/// Ordinary readers can ignore clustering information; providers share this one bit. +pub const FLAG_CLUSTERING_METADATA: u64 = 1 << 17; /// The first bit that is unknown as a feature flag pub const FLAG_UNKNOWN: u64 = 1 << 16; @@ -341,6 +345,7 @@ fn supported_flags_when( // Reserved, not implemented: see the flag's doc comment. mark_supported(&mut supported, FLAG_FRAGMENT_TREE, false); mark_supported(&mut supported, FLAG_INDEPENDENT_COVERING_FIELDS, false); + mark_supported(&mut supported, FLAG_CLUSTERING_METADATA, false); supported } @@ -531,6 +536,23 @@ mod tests { use super::*; use crate::format::BasePath; + #[test] + fn test_clustering_flag_fences_writes_without_blocking_reads() { + let mut manifest = empty_manifest(); + manifest.writer_feature_flags = FLAG_CLUSTERING_METADATA; + ensure_can_read_manifest(&manifest).unwrap(); + let error = ensure_can_write_manifest(&manifest).unwrap_err(); + assert!(matches!(error, Error::NotSupported { .. })); + assert!(error.to_string().contains("cannot be written")); + assert!(!can_read_dataset(FLAG_CLUSTERING_METADATA)); + assert!(!can_write_dataset(FLAG_CLUSTERING_METADATA)); + assert_eq!(FLAG_CLUSTERING_METADATA, 1 << 17); + // Reserving bit 17 must not accidentally enable the unassigned bit 16. + assert_eq!(FLAG_UNKNOWN, 1 << 16); + assert!(!can_read_dataset(FLAG_UNKNOWN)); + assert!(!can_write_dataset(FLAG_UNKNOWN)); + } + #[test] fn test_fragment_tree_flag_is_reserved_not_supported() { assert_eq!(FLAG_FRAGMENT_TREE, 4096); diff --git a/rust/lance-table/src/format/fragment.rs b/rust/lance-table/src/format/fragment.rs index b2fd3a850c6..22376d450f4 100644 --- a/rust/lance-table/src/format/fragment.rs +++ b/rust/lance-table/src/format/fragment.rs @@ -902,6 +902,7 @@ impl From<&Fragment> for pb::DataFragment { deletion_file, row_id_sequence, physical_rows: f.physical_rows.unwrap_or_default() as u64, + clustering: None, last_updated_at_version_sequence, created_at_version_sequence, } diff --git a/rust/lance-table/src/format/index.rs b/rust/lance-table/src/format/index.rs index 7f85b4112c8..1768a0e7db6 100644 --- a/rust/lance-table/src/format/index.rs +++ b/rust/lance-table/src/format/index.rs @@ -378,6 +378,7 @@ impl From<&IndexMetadata> for pb::IndexMetadata { .as_ref() .map(|details| details.as_ref().clone()), index_version: Some(idx.index_version), + clustering: None, created_at: idx.created_at.map(|dt| dt.timestamp_millis() as u64), base_id: idx.base_id, files, diff --git a/rust/lance-table/src/format/manifest.rs b/rust/lance-table/src/format/manifest.rs index cbadf3a44a6..f1878c0b36e 100644 --- a/rust/lance-table/src/format/manifest.rs +++ b/rust/lance-table/src/format/manifest.rs @@ -1143,6 +1143,7 @@ impl From<&Manifest> for pb::Manifest { .collect(), version: m.version, branch: m.branch.clone(), + clustering: None, writer_version: m .writer_version .as_ref() @@ -1261,6 +1262,80 @@ mod tests { use lance_core::datatypes::Field; use roaring::RoaringBitmap; + #[test] + fn clustering_proto_retains_provider_versions_and_unknown_metadata() { + let fragment_info = pb::ClusteringInfo { + provider: "org.example.cluster".into(), + version: 7, + metadata: HashMap::from([ + ("bucket".into(), "[0, 100)".into()), + ("unknown_option".into(), " opaque value: 東京 ".into()), + ("empty_option".into(), String::new()), + ]), + }; + let segment_info = pb::ClusteringInfo { + provider: "org.example.other".into(), + version: 9, + metadata: HashMap::from([("ranges".into(), "[0, 100, 200]".into())]), + }; + // Test the persisted messages only; runtime maintenance is not enabled yet. + let manifest = pb::Manifest { + writer_feature_flags: crate::feature_flags::FLAG_CLUSTERING_METADATA, + clustering: Some(pb::Clustering { + columns: vec![0, 1], + provider: "org.example.cluster".into(), + version: 8, + provider_metadata: HashMap::from([ + ("boundaries".into(), "[0, 100, 200]".into()), + ("unknown_option".into(), " opaque value: 東京 ".into()), + ]), + }), + fragments: vec![ + pb::DataFragment { + id: 1, + clustering: Some(fragment_info), + ..Default::default() + }, + pb::DataFragment { + id: 2, + clustering: Some(pb::ClusteringInfo { + provider: "org.example.cluster".into(), + version: 8, + metadata: HashMap::new(), + }), + ..Default::default() + }, + ], + ..Default::default() + }; + assert_eq!( + pb::Manifest::decode(manifest.encode_to_vec().as_slice()).unwrap(), + manifest + ); + assert_eq!(manifest.reader_feature_flags, 0); + let segments = pb::IndexSection { + indices: vec![pb::IndexMetadata { + name: "vectors".into(), + fields: vec![1], + clustering: Some(segment_info), + ..Default::default() + }], + }; + assert_eq!( + pb::IndexSection::decode(segments.encode_to_vec().as_slice()).unwrap(), + segments + ); + // Disabling the policy leaves existing fragment and segment information intact. + let disabled = pb::Manifest { + clustering: None, + ..manifest + }; + assert_eq!( + pb::Manifest::decode(disabled.encode_to_vec().as_slice()).unwrap(), + disabled + ); + } + /// A shallow clone points every local file at the parent through `base_id`. /// An overlay's data file lives in the parent too, so it needs the same /// stamp; without it the clone looks for the overlay under its own root. From 77bc8959af69072caa1023b6b837dd251f824513 Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:07:45 -0700 Subject: [PATCH 2/6] docs(format): describe clustering rules directly --- docs/src/format/table/clustering.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md index 3f3988f32ac..644241a28eb 100644 --- a/docs/src/format/table/clustering.md +++ b/docs/src/format/table/clustering.md @@ -1,7 +1,7 @@ # Clustering Metadata -Clustering organizes data to improve query pruning. Like Delta, Lance separates -the table's current policy from information about individual outputs. Providers +Clustering organizes data to improve query pruning. Lance records the table's +current policy separately from information about individual outputs. Providers choose the layout and maintenance logic. This proposal defines the format in `clustering.proto`; the writer feature remains unsupported until implemented. @@ -109,6 +109,3 @@ information are absent; disabling table clustering alone is insufficient. Providers share one flag, independent of layout version. FRI and write seeds keep their own feature requirements. This reservation does not enable clustering APIs or metadata-aware writes. - -Sources: [Delta clustered-table protocol](https://github.com/delta-io/delta/blob/ef3e91b509365cdc2b23c1afdbbab3b4d3828d63/PROTOCOL.md#clustered-table) -and [changing clustering columns](https://docs.delta.io/delta-clustering/#change-clustering-columns). From 2b120dbfc3ec203b084454b887b18ff990381b19 Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:08:35 -0700 Subject: [PATCH 3/6] docs(format): simplify clustering introduction --- docs/src/format/table/clustering.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md index 644241a28eb..f004db857ee 100644 --- a/docs/src/format/table/clustering.md +++ b/docs/src/format/table/clustering.md @@ -3,7 +3,7 @@ Clustering organizes data to improve query pruning. Lance records the table's current policy separately from information about individual outputs. Providers choose the layout and maintenance logic. This proposal defines the format in -`clustering.proto`; the writer feature remains unsupported until implemented. +`clustering.proto`. ## Stored Information From fdf460671c3f0baded13d8df85ed66813879a060 Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:13:48 -0700 Subject: [PATCH 4/6] docs(format): preserve existing clustering key hints --- docs/src/format/table/clustering.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md index f004db857ee..415a70c9099 100644 --- a/docs/src/format/table/clustering.md +++ b/docs/src/format/table/clustering.md @@ -34,9 +34,11 @@ must not be interpreted using a different current configuration. fragment and segment information. Historical reclustering is a separate operation. - Providers may take over other providers' fragments, recluster them, and write their own information on the outputs. Lance imposes no provider ownership boundary. -- When configuration is present, `columns` defines the clustering keys. Writers - clear the older `Field.unenforced_clustering_key_position` and - `Field.unenforced_clustering_key` hints to avoid conflicting definitions. +- When `Manifest.clustering` is present, implementations supporting this contract + use its `columns` as the clustering keys, taking precedence over existing + `Field.unenforced_clustering_key_position` and `Field.unenforced_clustering_key` + hints. Preserve those hints under their existing rules rather than clearing or + synchronizing them. Without the new configuration, their existing semantics apply. ## Clustering, Compaction, and Indexing From e18a488cf865a9cf5d7a04752df133db7e92656b Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:18:21 -0700 Subject: [PATCH 5/6] docs(format): clarify clustering maintenance and legacy hints --- docs/src/format/table/clustering.md | 50 +++++++++++++++++------------ 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md index 415a70c9099..0c10484b96d 100644 --- a/docs/src/format/table/clustering.md +++ b/docs/src/format/table/clustering.md @@ -1,9 +1,9 @@ # Clustering Metadata -Clustering organizes data to improve query pruning. Lance records the table's -current policy separately from information about individual outputs. Providers -choose the layout and maintenance logic. This proposal defines the format in -`clustering.proto`. +Clustering organizes data to improve query pruning. A **provider** is the +implementation that chooses the layout and how to maintain it. Lance records +the table's current policy separately from the layout information on fragments +and index segments. This proposal defines the format in `clustering.proto`. ## Stored Information @@ -34,11 +34,15 @@ must not be interpreted using a different current configuration. fragment and segment information. Historical reclustering is a separate operation. - Providers may take over other providers' fragments, recluster them, and write their own information on the outputs. Lance imposes no provider ownership boundary. -- When `Manifest.clustering` is present, implementations supporting this contract - use its `columns` as the clustering keys, taking precedence over existing - `Field.unenforced_clustering_key_position` and `Field.unenforced_clustering_key` - hints. Preserve those hints under their existing rules rather than clearing or - synchronizing them. Without the new configuration, their existing semantics apply. + +### Legacy Column-Level Hints + +Preserve `Field.unenforced_clustering_key_position` and +`Field.unenforced_clustering_key` under their existing rules. These are **legacy +column-level hints**, not fields to clear or synchronize with the new configuration. +When `Manifest.clustering` is present, supporting implementations use its `columns` +as the clustering keys instead. Otherwise, the legacy hints retain their existing +meaning. ## Clustering, Compaction, and Indexing @@ -56,25 +60,29 @@ Providers may reuse Lance's builders. Lance validates inputs against concurrent changes and commits the outputs and metadata atomically. Runtime interface signatures are outside this format contract. -Without an available provider compaction strategy, default compaction skips -clustered fragments and processes unclustered ones normally. An explicit ordinary -compaction may override this; its output is unmarked. Provider failures are -reported rather than silently switching strategies. Without a provider indexing -strategy, normal index building remains available, producing unmarked segments. -Fragments without clustering information use Lance's default compaction and -indexing unless a provider selects them for its own operations. +The presence of fragment clustering information determines default maintenance: + +- **Without clustering information:** use normal Lance compaction and indexing, + unless a provider selects the fragments for its own operations. +- **With clustering information:** use the provider's available maintenance + strategy. Without a compaction strategy, default compaction skips these + fragments. Explicit ordinary compaction may override this and produces + fragments without clustering information. Without an indexing strategy, normal + indexing remains available and produces segments without clustering information. + +Provider failures are reported, not silently retried with a different strategy. ## Ordinary Maintenance - Unrelated commits preserve configuration and unchanged fragments' and segments' information, including unknown metadata keys and values. -- Ordinary data rewrites do not inherit fragment clustering information. - Column replacement and overlays clear it even when the fragment ID remains - unchanged or only non-clustering columns change. No layout-impact analysis is needed. +- Ordinary data rewrites produce fragments without clustering information. + Column replacement and overlays clear it even when the fragment ID stays the + same or only non-clustering columns change. No layout-impact analysis is needed. - Deletion-vector-only changes preserve the original fragment's information. Providers must tolerate deleted rows and revalidate maintenance bookkeeping. -- Ordinary index merge, rebuild, and remap do not inherit information on the - rewritten segments. Data fragments' clustering information is unaffected. +- Ordinary index merge, rebuild, and remap produce segments without clustering + information. Data fragments' clustering information is unaffected. - Provider operations set their outputs' information. Metadata maps are not automatically copied or merged from inputs. From bf6c881273003a256cdc736a675a09094341d314 Mon Sep 17 00:00:00 2001 From: Lu Qiu Date: Tue, 13 Oct 2026 15:49:01 -0700 Subject: [PATCH 6/6] docs(format): explain clustering providers and maintenance rules --- docs/src/format/table/clustering.md | 233 +++++++++++++++------------- 1 file changed, 125 insertions(+), 108 deletions(-) diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md index 0c10484b96d..504917ae761 100644 --- a/docs/src/format/table/clustering.md +++ b/docs/src/format/table/clustering.md @@ -1,121 +1,138 @@ # Clustering Metadata -Clustering organizes data to improve query pruning. A **provider** is the -implementation that chooses the layout and how to maintain it. Lance records -the table's current policy separately from the layout information on fragments -and index segments. This proposal defines the format in `clustering.proto`. +Clustering puts rows with similar values close together. For example, grouping +rows by time helps a query for one day skip data from other days. Column +statistics tell queries which fragments, index segments, or smaller row groups +can be skipped. Writes can use the same information to reduce the data +they need to inspect. + +Lance does not choose a clustering algorithm or provide a default clustering +implementation. Instead, it defines metadata that different implementations can +use. Each implementation is called a **provider**. Contributors can build and +maintain their own providers. + +The provider decides how to organize data during writes. Readers do not need to +know that algorithm: they use column statistics to skip data. These statistics +can come from zone-level write seeds, a ZoneMap index, or fragment and segment +summaries built from the seeds. + +## What Lance Stores + +### Table Configuration + +`Manifest.clustering` stores the table's current clustering settings: + +- `columns`: the clustering columns, in order. Store their schema field IDs; + the list must be non-empty, with no duplicate or negative IDs, and every ID + must exist in the schema. +- `provider`: the provider's non-empty, case-sensitive name. +- `version`: a non-zero version assigned by the provider to these layout rules. +- `provider_metadata`: provider configuration as `map`. + +### Fragment and Index Segment Information + +Each fragment and index segment can also record how it was organized. +`DataFragment.clustering` and `IndexMetadata.clustering` store optional +`ClusteringInfo`: + +- `provider`: who organized it. +- `version`: which version of that provider's layout rules was used. +- `metadata`: extra layout details as `map`. + +The provider defines the map's keys and string values, such as bucket, range, or +group IDs. Maps may be empty. A segment's metadata describes its own grouping; +it does not change the metadata on its input fragments. + +## What Providers Do + +Providers decide which fragments to process together, do the work, and write +metadata for the results: + +- **Clustering:** choose fragments, reorganize their rows, and write new + fragments with clustering metadata. +- **Compaction:** choose fragments to compact together and write the compacted + fragments with clustering metadata. +- **Indexing:** choose groups for each index, build one segment per group, and + write segment clustering metadata. Different indices can group the same + fragments differently. + +Providers can reuse Lance's builders. Lance checks the inputs and conflicting +concurrent changes, then commits the new files and metadata together. The +provider API is outside this format proposal. + +### When No Provider Strategy Is Available + +- **Fragments without clustering metadata:** normal Lance compaction and + indexing apply, unless a provider chooses to process them. +- **Compaction of fragments with clustering metadata:** use the provider's + strategy if available. Otherwise, skip them by default. Users may explicitly + request normal compaction; its new fragments have no clustering metadata. +- **Indexing of fragments with clustering metadata:** use the provider's strategy + if available. Otherwise, normal indexing can still run; its new segments have + no clustering metadata. + +If a provider fails, return an error. Do not silently switch to normal maintenance. + +## What Happens During Maintenance + +- **Nothing changed:** keep the table settings and metadata on unchanged + fragments and segments, including keys and values the writer does not recognize. +- **Normal data rewrite:** clear clustering metadata on the rewritten fragments. + Column replacement, refresh, and overlays follow this rule even if the fragment + ID stays the same or the changed column is not a clustering key. There is no + need to decide whether the layout was affected. +- **Only deletion vectors changed:** keep the fragment's clustering metadata. + Providers must account for deleted rows when planning later maintenance. +- **Index merge:** the provider decides how to merge and writes the new segments' + clustering metadata. A normal merge without a provider strategy produces + segments with no clustering metadata. +- **Index remap or rebuild:** keep the segment's clustering metadata. +- **New provider-created layout:** the provider writes its own metadata. Lance + does not copy or combine the input maps automatically. + +Index maintenance does not change the data fragments' clustering metadata. + +## How Readers Benefit + +Providers should compare clustering values using Lance's defined Arrow Rust +ordering rules, because readers use those rules for column statistics. Sorting +values differently can make the min/max ranges wider and reduce pruning. It does +not change the reader's rules. -## Stored Information +Clustering writes collect zone-level write seeds for clustering columns using +the same rules. Readers can use zone statistics or combine them into fragment +and segment summaries. This does not require a particular statistics index. If +statistics are missing or unusable, keep the data as a query candidate. -`Manifest.clustering` holds the current `Clustering` configuration: +Readers can ignore clustering metadata. A provider's metadata is not proof that +data can be skipped: readers still need valid statistics and index coverage. +FRI address translation and deletion handling follow their own rules. Clustering +metadata must not add hidden requirements for correct reads or normal writes. -- `columns`: an ordered, non-empty list of distinct, non-negative schema field - IDs for existing clustering columns. -- `provider`: a non-empty, case-sensitive implementation name. -- `version`: a non-zero, provider-assigned layout/configuration identifier. -- `provider_metadata`: `map` containing provider configuration. +## Reader and Writer Compatibility -`DataFragment.clustering` and `IndexMetadata.clustering` each hold optional -`ClusteringInfo`: `provider`, `version`, and `metadata: map`. -Providers define keys and string encodings, such as bucket, range, or group -identifiers. Empty maps are valid. A segment's information describes its own -grouping; it does not relabel its input fragments. +Set `FLAG_CLUSTERING_METADATA` (`1 << 17`) in a snapshot's `writer_feature_flags` +if it contains any of the following: -`version` is not a dataset version or software release. Within a table, including -branches, `(provider, version)` must keep the same layout meaning. Layout changes -use a new version; providers manage any history they need. An unknown old version -must not be interpreted using a different current configuration. +- Table clustering configuration. +- Fragment clustering information, including records in a fragment metadata tree. +- Index segment clustering information, including records in the index section. -## Main Rules +Readers need no clustering flag because they can ignore this metadata. Writers +that do not support the feature must refuse writes. Writers that do support it +follow the rules above; they do not need to understand every provider. -- Enabling clustering does not require every append to cluster its output or - impose a deadline for organizing existing data. -- Changing or removing configuration does not rewrite history or erase existing - fragment and segment information. Historical reclustering is a separate operation. -- Providers may take over other providers' fragments, recluster them, and write - their own information on the outputs. Lance imposes no provider ownership boundary. +Clear the flag only when all three are absent. Removing the table configuration +alone is not enough. All providers and layout versions use this one flag. FRI +and write seeds have separate feature requirements. Reserving this flag does +not implement clustering APIs or metadata-aware writes. ### Legacy Column-Level Hints -Preserve `Field.unenforced_clustering_key_position` and -`Field.unenforced_clustering_key` under their existing rules. These are **legacy -column-level hints**, not fields to clear or synchronize with the new configuration. -When `Manifest.clustering` is present, supporting implementations use its `columns` -as the clustering keys instead. Otherwise, the legacy hints retain their existing -meaning. - -## Clustering, Compaction, and Indexing - -Providers choose the fragments to process and return groups of fragments: - -- **Clustering:** the provider chooses which fragments to recluster together, - reorganizes their rows, and writes its information on the new fragments. -- **Compaction:** the provider compacts each group and writes clustering - information for the new fragments. -- **Indexing:** each index has its own groups. The provider builds each segment - and writes its clustering information; different index types may use different - grouping sizes over the same fragment layout. - -Providers may reuse Lance's builders. Lance validates inputs against concurrent -changes and commits the outputs and metadata atomically. Runtime interface -signatures are outside this format contract. - -The presence of fragment clustering information determines default maintenance: - -- **Without clustering information:** use normal Lance compaction and indexing, - unless a provider selects the fragments for its own operations. -- **With clustering information:** use the provider's available maintenance - strategy. Without a compaction strategy, default compaction skips these - fragments. Explicit ordinary compaction may override this and produces - fragments without clustering information. Without an indexing strategy, normal - indexing remains available and produces segments without clustering information. - -Provider failures are reported, not silently retried with a different strategy. - -## Ordinary Maintenance - -- Unrelated commits preserve configuration and unchanged fragments' and - segments' information, including unknown metadata keys and values. -- Ordinary data rewrites produce fragments without clustering information. - Column replacement and overlays clear it even when the fragment ID stays the - same or only non-clustering columns change. No layout-impact analysis is needed. -- Deletion-vector-only changes preserve the original fragment's information. - Providers must tolerate deleted rows and revalidate maintenance bookkeeping. -- Ordinary index merge, rebuild, and remap produce segments without clustering - information. Data fragments' clustering information is unaffected. -- Provider operations set their outputs' information. Metadata maps are not - automatically copied or merged from inputs. - -Dropping a clustering field or changing its type removes the current configuration. -Renaming while retaining field IDs and types preserves it. Overwrite may retain -configuration only when its field IDs and types remain compatible. Clearing output -information does not otherwise disable the policy. Restore restores the selected -snapshot's information; clone or import preserves layout identity or clears -conflicting claims. Metadata paths acquire no file-retention or clone semantics. - -## Statistics and Reads +`Field.unenforced_clustering_key_position` and `Field.unenforced_clustering_key` +are **legacy column-level hints**. Keep their existing rules. Do not remove them +or change them to match the new configuration. -Clustering writes collect zone-level write seeds for clustering columns using -Lance's defined Arrow Rust comparison semantics. Consumers may use zones directly -or aggregate them into fragment or segment statistics; no particular statistics -index is required. Missing or unusable statistics keep the data as a query candidate. - -Readers may ignore clustering information. Provider metadata alone cannot prove -row order or justify pruning. Statistics, index coverage, FRI translation, and -deletions retain their own contracts. Metadata cannot impose hidden requirements -for correct queries or ordinary writes. - -## Compatibility - -A snapshot referencing table configuration, any fragment clustering information, -or any index segment clustering information must set `FLAG_CLUSTERING_METADATA` -(`1 << 17`) in `writer_feature_flags`, not `reader_feature_flags`. This includes -fragments in a fragment metadata tree and indices in the index section. - -Older writers must refuse to write; readers need no clustering capability. -Supporting writers preserve or clear information using the rules above without -understanding every provider. Clear the flag only when all three kinds of -information are absent; disabling table clustering alone is insufficient. -Providers share one flag, independent of layout version. FRI and write seeds -keep their own feature requirements. This reservation does not enable clustering -APIs or metadata-aware writes. +When `Manifest.clustering` is present, implementations supporting this proposal +use its `columns` as the clustering keys instead of the legacy hints. Otherwise, +the hints keep their existing meaning.