diff --git a/docs/src/format/table/.pages b/docs/src/format/table/.pages index 064c080e5f4..e1fe5b007c3 100644 --- a/docs/src/format/table/.pages +++ b/docs/src/format/table/.pages @@ -7,5 +7,6 @@ nav: - Branch & Tag: branch_tag.md - Row ID & Lineage: row_id_lineage.md - Data Overlay Files: data_overlay_file.md + - Clustering Metadata: clustering.md - Fragment Metadata Tree: fragment_metadata.md - MemTable & WAL: mem_wal.md diff --git a/docs/src/format/table/clustering.md b/docs/src/format/table/clustering.md new file mode 100644 index 00000000000..504917ae761 --- /dev/null +++ b/docs/src/format/table/clustering.md @@ -0,0 +1,138 @@ +# Clustering Metadata + +Clustering puts rows with similar values close together. For example, grouping +rows by time helps a query for one day skip data from other days. Column +statistics tell queries which fragments, index segments, or smaller row groups +can be skipped. Writes can use the same information to reduce the data +they need to inspect. + +Lance does not choose a clustering algorithm or provide a default clustering +implementation. Instead, it defines metadata that different implementations can +use. Each implementation is called a **provider**. Contributors can build and +maintain their own providers. + +The provider decides how to organize data during writes. Readers do not need to +know that algorithm: they use column statistics to skip data. These statistics +can come from zone-level write seeds, a ZoneMap index, or fragment and segment +summaries built from the seeds. + +## What Lance Stores + +### Table Configuration + +`Manifest.clustering` stores the table's current clustering settings: + +- `columns`: the clustering columns, in order. Store their schema field IDs; + the list must be non-empty, with no duplicate or negative IDs, and every ID + must exist in the schema. +- `provider`: the provider's non-empty, case-sensitive name. +- `version`: a non-zero version assigned by the provider to these layout rules. +- `provider_metadata`: provider configuration as `map`. + +### Fragment and Index Segment Information + +Each fragment and index segment can also record how it was organized. +`DataFragment.clustering` and `IndexMetadata.clustering` store optional +`ClusteringInfo`: + +- `provider`: who organized it. +- `version`: which version of that provider's layout rules was used. +- `metadata`: extra layout details as `map`. + +The provider defines the map's keys and string values, such as bucket, range, or +group IDs. Maps may be empty. A segment's metadata describes its own grouping; +it does not change the metadata on its input fragments. + +## What Providers Do + +Providers decide which fragments to process together, do the work, and write +metadata for the results: + +- **Clustering:** choose fragments, reorganize their rows, and write new + fragments with clustering metadata. +- **Compaction:** choose fragments to compact together and write the compacted + fragments with clustering metadata. +- **Indexing:** choose groups for each index, build one segment per group, and + write segment clustering metadata. Different indices can group the same + fragments differently. + +Providers can reuse Lance's builders. Lance checks the inputs and conflicting +concurrent changes, then commits the new files and metadata together. The +provider API is outside this format proposal. + +### When No Provider Strategy Is Available + +- **Fragments without clustering metadata:** normal Lance compaction and + indexing apply, unless a provider chooses to process them. +- **Compaction of fragments with clustering metadata:** use the provider's + strategy if available. Otherwise, skip them by default. Users may explicitly + request normal compaction; its new fragments have no clustering metadata. +- **Indexing of fragments with clustering metadata:** use the provider's strategy + if available. Otherwise, normal indexing can still run; its new segments have + no clustering metadata. + +If a provider fails, return an error. Do not silently switch to normal maintenance. + +## What Happens During Maintenance + +- **Nothing changed:** keep the table settings and metadata on unchanged + fragments and segments, including keys and values the writer does not recognize. +- **Normal data rewrite:** clear clustering metadata on the rewritten fragments. + Column replacement, refresh, and overlays follow this rule even if the fragment + ID stays the same or the changed column is not a clustering key. There is no + need to decide whether the layout was affected. +- **Only deletion vectors changed:** keep the fragment's clustering metadata. + Providers must account for deleted rows when planning later maintenance. +- **Index merge:** the provider decides how to merge and writes the new segments' + clustering metadata. A normal merge without a provider strategy produces + segments with no clustering metadata. +- **Index remap or rebuild:** keep the segment's clustering metadata. +- **New provider-created layout:** the provider writes its own metadata. Lance + does not copy or combine the input maps automatically. + +Index maintenance does not change the data fragments' clustering metadata. + +## How Readers Benefit + +Providers should compare clustering values using Lance's defined Arrow Rust +ordering rules, because readers use those rules for column statistics. Sorting +values differently can make the min/max ranges wider and reduce pruning. It does +not change the reader's rules. + +Clustering writes collect zone-level write seeds for clustering columns using +the same rules. Readers can use zone statistics or combine them into fragment +and segment summaries. This does not require a particular statistics index. If +statistics are missing or unusable, keep the data as a query candidate. + +Readers can ignore clustering metadata. A provider's metadata is not proof that +data can be skipped: readers still need valid statistics and index coverage. +FRI address translation and deletion handling follow their own rules. Clustering +metadata must not add hidden requirements for correct reads or normal writes. + +## Reader and Writer Compatibility + +Set `FLAG_CLUSTERING_METADATA` (`1 << 17`) in a snapshot's `writer_feature_flags` +if it contains any of the following: + +- Table clustering configuration. +- Fragment clustering information, including records in a fragment metadata tree. +- Index segment clustering information, including records in the index section. + +Readers need no clustering flag because they can ignore this metadata. Writers +that do not support the feature must refuse writes. Writers that do support it +follow the rules above; they do not need to understand every provider. + +Clear the flag only when all three are absent. Removing the table configuration +alone is not enough. All providers and layout versions use this one flag. FRI +and write seeds have separate feature requirements. Reserving this flag does +not implement clustering APIs or metadata-aware writes. + +### Legacy Column-Level Hints + +`Field.unenforced_clustering_key_position` and `Field.unenforced_clustering_key` +are **legacy column-level hints**. Keep their existing rules. Do not remove them +or change them to match the new configuration. + +When `Manifest.clustering` is present, implementations supporting this proposal +use its `columns` as the clustering keys instead of the legacy hints. Otherwise, +the hints keep their existing meaning. diff --git a/docs/src/format/table/versioning.md b/docs/src/format/table/versioning.md index e4d77b9f273..c38cdae5c75 100644 --- a/docs/src/format/table/versioning.md +++ b/docs/src/format/table/versioning.md @@ -40,8 +40,10 @@ they should return an "unsupported" error on any read or write operation. | `2^14` | `FLAG_MANAGED_BLOBS` | Yes | Yes | Blob descriptors independently address owned objects. Readers resolve the descriptor's base and path; cleanup preserves objects referenced by protected snapshots, independently of the original data file. Both bits remain set on later versions, including restores. | | `2^15` | `FLAG_NON_REUSABLE_FIELD_IDS` | No | Yes | The manifest sets `max_allocated_field_id`, and a writer must assign new field IDs above it. Before activation, all clients that can write to the dataset must enforce writer feature flags. See [Field IDs](schema.md#field-ids). | +| `2^17` | `FLAG_CLUSTERING_METADATA` | No | Yes | The snapshot references [clustering configuration or fragment/index segment information](clustering.md). Reserved, unsupported until writers implement its maintenance rules. Readers may ignore clustering metadata. | + -Bits at positions 16 and above (`2^16`, `2^17`, and so on) are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version and Managed Blob reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. +Bit 17 is reserved for clustering metadata but remains unsupported. Other bits at positions 16 and above are unknown. Unknown or unsupported flags cause implementations to reject the corresponding read or write operation with an "unsupported" error. The paired mixed-version and Managed Blob reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. Publishing a new data file containing a Blob v2 field activates the Managed Blob capability, including files containing only inline values. Metadata-only updates and deletion vectors on existing files do not activate it. Once activated, removing Blob fields or restoring an older snapshot does not clear it. Before activation, all maintenance clients must support Managed Blobs: the flag cannot revoke old handles or prevent older clients from running cleanup through an unflagged historical snapshot. See [client compatibility](../../guide/blob.md#managed-objects-and-client-compatibility). diff --git a/protos/clustering.proto b/protos/clustering.proto new file mode 100644 index 00000000000..0c9d97ba62e --- /dev/null +++ b/protos/clustering.proto @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +syntax = "proto3"; + +package lance.table; + +// Current table-level clustering configuration. Providers manage layout history. +message Clustering { + // Ordered, non-empty list of distinct, existing schema field IDs. + repeated int32 columns = 1; + // Non-empty, case-sensitive clustering implementation name. + string provider = 2; + // Non-zero layout/configuration identifier, not a dataset or software version. + uint64 version = 3; + /* Provider-defined configuration. Empty is valid. Providers define keys and + * value encodings; unrelated writes preserve unknown keys and values. + */ + map provider_metadata = 4; +} + +// Clustering information shared by fragments and index segments. +message ClusteringInfo { + // Non-empty, case-sensitive implementation name that organized this output. + string provider = 1; + // Non-zero identifier of the provider's layout/configuration for this output. + uint64 version = 2; + /* Provider-defined information, such as group, bucket, or range identifiers. + * Empty is valid. It is not proof of index coverage, statistics, or row order. + * Ordinary rewrites do not inherit this information; provider operations set + * it for their outputs. Unchanged objects preserve unknown keys and values. + */ + map metadata = 3; +} diff --git a/protos/fragment_metadata.proto b/protos/fragment_metadata.proto index 63e2651d1e8..861bb103018 100644 --- a/protos/fragment_metadata.proto +++ b/protos/fragment_metadata.proto @@ -5,6 +5,8 @@ syntax = "proto3"; package lance.table; +import "clustering.proto"; + /* A DataFragment is a set of files which represent the different columns of the same * rows. If column exists in the schema of a dataset, but the file for that column does * not exist within a DataFragment of that dataset, that column consists entirely of @@ -85,6 +87,12 @@ message DataFragment { * `deletion_file.num_deleted_rows` from this value. */ uint64 physical_rows = 4; + + /* Optional provider, layout version, and metadata for this fragment. + * Ordinary rewrites do not inherit it; deletion-vector-only changes preserve + * it. Absence makes no clustering claim. Requires writer flag 1 << 17. + */ + ClusteringInfo clustering = 15; } message DataFile { diff --git a/protos/table.proto b/protos/table.proto index 2370324a346..5df8b053604 100644 --- a/protos/table.proto +++ b/protos/table.proto @@ -8,6 +8,7 @@ package lance.table; import "google/protobuf/any.proto"; import "google/protobuf/timestamp.proto"; import "file.proto"; +import "clustering.proto"; import public "fragment_metadata.proto"; /* @@ -290,6 +291,12 @@ message Manifest { * both be set or both be absent. A mismatch is invalid. */ optional int32 max_allocated_field_id = 24; + + /* Optional current clustering policy. Absence disables new clustering work + * without clearing fragment or index segment information. Requires the + * clustering writer feature flag (1 << 17); no clustering reader flag. + */ + Clustering clustering = 25; } // Manifest // external dataset base path @@ -415,6 +422,12 @@ message IndexMetadata { * withdraw the payload while leaving this declaration in place. */ repeated int32 covering_fields = 11; + + /* Optional clustering information for this segment's grouping. Ordinary + * merge, rebuild, and remap do not inherit it; provider operations set it for + * their outputs. Absence makes no clustering claim. Requires writer flag 1 << 17. + */ + ClusteringInfo clustering = 12; } // Metadata about a single file within an index segment. diff --git a/rust/lance-table/benches/manifest_intern.rs b/rust/lance-table/benches/manifest_intern.rs index 81bd57c1a22..358b42a6a51 100644 --- a/rust/lance-table/benches/manifest_intern.rs +++ b/rust/lance-table/benches/manifest_intern.rs @@ -63,6 +63,7 @@ fn make_uniform_pb_fragments(n: u64, num_fields: usize) -> Vec deletion_file: None, row_id_sequence: None, physical_rows: 1000, + clustering: None, last_updated_at_version_sequence: Some( pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions( version_bytes.clone(), @@ -140,6 +141,7 @@ fn make_diverse_pb_fragments( deletion_file: None, row_id_sequence: None, physical_rows: 1000, + clustering: None, last_updated_at_version_sequence: Some( pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions( version_payloads[version_idx].clone(), diff --git a/rust/lance-table/src/feature_flags.rs b/rust/lance-table/src/feature_flags.rs index 435a2a7fe21..2f2390a71c7 100644 --- a/rust/lance-table/src/feature_flags.rs +++ b/rust/lance-table/src/feature_flags.rs @@ -109,6 +109,10 @@ pub const MANAGED_BLOBS_META_KEY: &str = "lance.blob.managed"; /// Writers must understand this allocation contract. It does not change how /// readers interpret the schema or data files. pub const FLAG_NON_REUSABLE_FIELD_IDS: u64 = 1 << 15; +/// Writer-only protection for table, fragment, and index segment clustering metadata. +/// Reserved beyond the supported range until writers implement its maintenance rules. +/// Ordinary readers can ignore clustering information; providers share this one bit. +pub const FLAG_CLUSTERING_METADATA: u64 = 1 << 17; /// The first bit that is unknown as a feature flag pub const FLAG_UNKNOWN: u64 = 1 << 16; @@ -341,6 +345,7 @@ fn supported_flags_when( // Reserved, not implemented: see the flag's doc comment. mark_supported(&mut supported, FLAG_FRAGMENT_TREE, false); mark_supported(&mut supported, FLAG_INDEPENDENT_COVERING_FIELDS, false); + mark_supported(&mut supported, FLAG_CLUSTERING_METADATA, false); supported } @@ -531,6 +536,23 @@ mod tests { use super::*; use crate::format::BasePath; + #[test] + fn test_clustering_flag_fences_writes_without_blocking_reads() { + let mut manifest = empty_manifest(); + manifest.writer_feature_flags = FLAG_CLUSTERING_METADATA; + ensure_can_read_manifest(&manifest).unwrap(); + let error = ensure_can_write_manifest(&manifest).unwrap_err(); + assert!(matches!(error, Error::NotSupported { .. })); + assert!(error.to_string().contains("cannot be written")); + assert!(!can_read_dataset(FLAG_CLUSTERING_METADATA)); + assert!(!can_write_dataset(FLAG_CLUSTERING_METADATA)); + assert_eq!(FLAG_CLUSTERING_METADATA, 1 << 17); + // Reserving bit 17 must not accidentally enable the unassigned bit 16. + assert_eq!(FLAG_UNKNOWN, 1 << 16); + assert!(!can_read_dataset(FLAG_UNKNOWN)); + assert!(!can_write_dataset(FLAG_UNKNOWN)); + } + #[test] fn test_fragment_tree_flag_is_reserved_not_supported() { assert_eq!(FLAG_FRAGMENT_TREE, 4096); diff --git a/rust/lance-table/src/format/fragment.rs b/rust/lance-table/src/format/fragment.rs index b2fd3a850c6..22376d450f4 100644 --- a/rust/lance-table/src/format/fragment.rs +++ b/rust/lance-table/src/format/fragment.rs @@ -902,6 +902,7 @@ impl From<&Fragment> for pb::DataFragment { deletion_file, row_id_sequence, physical_rows: f.physical_rows.unwrap_or_default() as u64, + clustering: None, last_updated_at_version_sequence, created_at_version_sequence, } diff --git a/rust/lance-table/src/format/index.rs b/rust/lance-table/src/format/index.rs index 7f85b4112c8..1768a0e7db6 100644 --- a/rust/lance-table/src/format/index.rs +++ b/rust/lance-table/src/format/index.rs @@ -378,6 +378,7 @@ impl From<&IndexMetadata> for pb::IndexMetadata { .as_ref() .map(|details| details.as_ref().clone()), index_version: Some(idx.index_version), + clustering: None, created_at: idx.created_at.map(|dt| dt.timestamp_millis() as u64), base_id: idx.base_id, files, diff --git a/rust/lance-table/src/format/manifest.rs b/rust/lance-table/src/format/manifest.rs index cbadf3a44a6..f1878c0b36e 100644 --- a/rust/lance-table/src/format/manifest.rs +++ b/rust/lance-table/src/format/manifest.rs @@ -1143,6 +1143,7 @@ impl From<&Manifest> for pb::Manifest { .collect(), version: m.version, branch: m.branch.clone(), + clustering: None, writer_version: m .writer_version .as_ref() @@ -1261,6 +1262,80 @@ mod tests { use lance_core::datatypes::Field; use roaring::RoaringBitmap; + #[test] + fn clustering_proto_retains_provider_versions_and_unknown_metadata() { + let fragment_info = pb::ClusteringInfo { + provider: "org.example.cluster".into(), + version: 7, + metadata: HashMap::from([ + ("bucket".into(), "[0, 100)".into()), + ("unknown_option".into(), " opaque value: 東京 ".into()), + ("empty_option".into(), String::new()), + ]), + }; + let segment_info = pb::ClusteringInfo { + provider: "org.example.other".into(), + version: 9, + metadata: HashMap::from([("ranges".into(), "[0, 100, 200]".into())]), + }; + // Test the persisted messages only; runtime maintenance is not enabled yet. + let manifest = pb::Manifest { + writer_feature_flags: crate::feature_flags::FLAG_CLUSTERING_METADATA, + clustering: Some(pb::Clustering { + columns: vec![0, 1], + provider: "org.example.cluster".into(), + version: 8, + provider_metadata: HashMap::from([ + ("boundaries".into(), "[0, 100, 200]".into()), + ("unknown_option".into(), " opaque value: 東京 ".into()), + ]), + }), + fragments: vec![ + pb::DataFragment { + id: 1, + clustering: Some(fragment_info), + ..Default::default() + }, + pb::DataFragment { + id: 2, + clustering: Some(pb::ClusteringInfo { + provider: "org.example.cluster".into(), + version: 8, + metadata: HashMap::new(), + }), + ..Default::default() + }, + ], + ..Default::default() + }; + assert_eq!( + pb::Manifest::decode(manifest.encode_to_vec().as_slice()).unwrap(), + manifest + ); + assert_eq!(manifest.reader_feature_flags, 0); + let segments = pb::IndexSection { + indices: vec![pb::IndexMetadata { + name: "vectors".into(), + fields: vec![1], + clustering: Some(segment_info), + ..Default::default() + }], + }; + assert_eq!( + pb::IndexSection::decode(segments.encode_to_vec().as_slice()).unwrap(), + segments + ); + // Disabling the policy leaves existing fragment and segment information intact. + let disabled = pb::Manifest { + clustering: None, + ..manifest + }; + assert_eq!( + pb::Manifest::decode(disabled.encode_to_vec().as_slice()).unwrap(), + disabled + ); + } + /// A shallow clone points every local file at the parent through `base_id`. /// An overlay's data file lives in the parent too, so it needs the same /// stamp; without it the clone looks for the overlay under its own root.