[DRAFT] feat(format): define provider-neutral clustering metadata

docs/src/format/table/.pages

@@ -6,5 +6,6 @@ nav:
   - Layout: layout.md
   - Branch & Tag: branch_tag.md
   - Row ID & Lineage: row_id_lineage.md
+  - Clustering Providers: clustering.md
   - Data Overlay Files: data_overlay_file.md
   - MemTable & WAL: mem_wal.md

docs/src/format/table/clustering.md

@@ -0,0 +1,166 @@
+# Clustering Providers
+
+Clustering describes a physical layout, not different logical table contents.
+The format identifies the provider and configuration used to organize fragments;
+it does not prescribe an algorithm, boundaries, or maintenance schedule.
+
+The definitions are `Clustering` and `FragmentClustering` in `table.proto`.
+This feature is reserved but unsupported until its writer preservation and
+invalidation rules are implemented.
+
+## Table Configuration
+
+`Manifest.clustering` holds the current configuration:
+
+- `columns`: an ordered, non-empty list of distinct, non-negative schema field
+  IDs that exist in the current schema. These identify all columns on which the
+  layout depends.
+- `provider`: a non-empty, case-sensitive implementation name. Namespaced names
+  are recommended to avoid collisions.
+- `version`: a non-zero `uint64` identifying a provider-defined layout or
+  configuration, not a dataset version or software release.
+- `provider_metadata`: a `map<string, string>`, like `Manifest.config`. An empty
+  map means no provider-specific configuration. The provider defines the keys,
+  value encoding, schema versioning, and compatibility. Structured values may be
+  serialized as JSON strings; Lance does not interpret or normalize them.
+
+Absence of table configuration disables new clustering work but does not clear
+existing fragment markers. Appends need not be clustered even when a
+configuration is present.
+
+The pair `(provider, version)` must not be reassigned to a different layout
+definition within a table. Providers allocate versions, including on divergent
+branches. No numerical ordering or compatibility is implied by the version.
+Changing the layout definition selects a new version; auxiliary history or
+maintenance information in the payload may evolve without redefining that layout.
+
+Lance does not maintain a declaration list or resolve historical versions.
+Providers manage any history they require in their own metadata. Switching the
+current provider or version does not relabel or rewrite old fragments. A marker
+may name an older version or another provider. If its configuration is unavailable
+or unsupported, it must not be interpreted using the current configuration.
+
+Provider metadata is inline. Paths in metadata values acquire no file-retention
+or clone semantics; external artifacts require a separate lifecycle contract.
+Providers must tolerate ordinary writers replacing fragments or clearing markers
+without updating opaque bookkeeping. Such bookkeeping is not authoritative table state.
+
+When `Manifest.clustering` is present, its `columns` are the source of clustering
+keys. Writers must clear `Field.unenforced_clustering_key_position` and the older
+`Field.unenforced_clustering_key` hints rather than maintain a second copy. When
+configuration is absent, existing hints retain their prior meaning and do not
+imply fragment provenance.
+
+## Fragment Markers
+
+`DataFragment.clustering` contains only `provider` and `version`, with the same
+validity rules as the table configuration. Absence makes no clustering claim.
+A marker identifies the layout under which the fragment was organized; it does
+not prove sort order, disjoint ranges, size, or query coverage. Fragments carry
+no opaque payload or arbitrary key-value map.
+
+Only a writer that understands the relevant configuration may assign its marker.
+The layout `version` already identifies a configuration generation; there is no
+separate generation field. Rewrite groups, buckets, and their fragment membership
+are provider-specific concepts, not public fragment fields. Providers that need
+them manage their records in table-level `provider_metadata`; no common keys or
+value schemas are prescribed. These records must be revalidated against live
+fragments and their markers before use, since ordinary writers do not maintain
+opaque provider bookkeeping. A marker may remain when the current table
+configuration is absent or different.
+
+By default, clustering maintenance selects unstamped fragments or fragments
+belonging to the same provider, or an implementation it explicitly supersedes.
+Other providers' fragments require an explicit user takeover. Sharing a provider
+name does not make all its versions interpretable by every implementation.
+An unsupported configuration must not be executed or silently substituted.
+
+These rules govern clustering maintenance, not ordinary table writes. Missing a
+provider must not by itself prohibit append, update, delete, or compaction.
+
+## Writer Requirements
+
+Writers supporting the common feature preserve unknown provider names, versions,
+and all metadata keys and string values on unrelated commits. Not understanding
+a provider does not authorize changing its configuration. It also does not require
+stopping ordinary writes: the generic rules below suffice.
+
+- Append preserves existing metadata. New fragments are unstamped unless a
+  provider verifies their layout.
+- An unchanged fragment retains its marker. Deletion-only changes may retain it:
+  layout claims must remain valid for a subset of rows at unchanged offsets,
+  without promising current size or clustering quality.
+- A new fragment from a generic split, merge, compaction, or row rewrite does
+  not inherit a marker. A provider-aware writer may verify and assign one.
+- In-place column replacement or overlays changing a clustering column clear
+  the affected fragment's marker. If the marker matches the current table pair,
+  its columns identify those dependencies. For any other pair, a generic writer
+  cannot infer historical dependencies and clears the marker on any value change.
+  A provider-aware writer may retain it only after verifying layout validity.
+- A rename retaining field IDs and types preserves markers. Dropping a field,
+  changing its type, or changing a containing nested type clears affected
+  current-layout markers and removes the table configuration if its columns are
+  affected. For historical markers with unknown dependencies, such schema changes
+  conservatively clear the markers. Missing a provider must not block the change.
+- Overwrite discards replaced fragments' markers. It may retain table
+  configuration only if its field IDs and types remain compatible; otherwise it
+  removes that configuration. Historical columns must not be rebound by name.
+
+Clearing a fragment marker alone does not remove table configuration or modify
+its opaque payload. Explicit configuration changes and schema invalidation above
+are distinct from unrelated commits that must preserve the configuration.
+
+Each commit must publish valid configuration and markers atomically and revalidate
+them against concurrent changes. The provider, version, columns, and metadata
+form one configuration; the map shape does not permit generic per-key merging of
+concurrent configuration changes. A rewrite may publish unstamped output and mark
+it in a later commit only after validating that the output has not changed.
+Restore restores the selected snapshot's configuration and markers together.
+Clone or import must preserve layout identity to retain markers; conflicting
+definitions for the same pair must be rejected or the imported markers cleared.
+
+## Readers and Statistics
+
+Readers may ignore clustering metadata and scan normally. They must not prune
+data or assume ordering based on a marker alone. Index coverage, row-address
+translation, and deletions keep their own contracts and feature requirements.
+
+Statistics producers and consumers must share value-comparison semantics,
+including strings, nulls, NaNs, and typed bounds. Provider normalization cannot
+silently change these semantics. Custom statistics require a reader or index
+plugin that understands them; otherwise that optimization must not be used.
+Missing or incomplete statistics cannot justify skipping data. No particular
+statistics format or index is required by this feature.
+
+## Compatibility
+
+A manifest containing table configuration or any fragment marker must set
+`FLAG_CLUSTERING_METADATA` (`1 << 12`) in `writer_feature_flags`, not in
+`reader_feature_flags`. This single bit protects the common metadata container
+and its maintenance rules. Older writers that could discard the new fields
+must refuse to write. It may be cleared only when configuration and all markers
+are absent.
+
+A writer supporting the common feature can preserve an unknown provider and
+perform ordinary writes using the rules above. Requests to execute unsupported
+provider configurations are rejected without prohibiting other operations.
+
+Adding a provider, changing layout versions, or evolving a provider's payload
+does not allocate another Lance feature bit. Providers do not own global reader
+or writer bits. Layout versions identify configurations, not an ordered
+compatibility threshold; interpretation belongs to the provider. Payload schema
+compatibility can be expressed through provider-defined metadata keys and values.
+
+A provider cannot use opaque metadata to impose extra correctness requirements
+on ordinary readers or writers. Behavior that cannot safely be ignored,
+preserved, or invalidated under this contract requires a separate table feature,
+not just a provider version bump.
+
+## Relationship to Delta
+
+Delta's [clustered-table protocol](https://github.com/delta-io/delta/blob/ef3e91b509365cdc2b23c1afdbbab3b4d3828d63/PROTOCOL.md#clustered-table)
+uses shared writer features rather than a feature for each clustering provider.
+Its [domain metadata contract](https://github.com/delta-io/delta/blob/ef3e91b509365cdc2b23c1afdbbab3b4d3828d63/PROTOCOL.md#domain-metadata)
+requires preservation of unknown domains, while additional correctness obligations
+require feature negotiation. This proposal follows that separation and provider
+ownership principle without adopting Delta's JSON actions or general file tags.

docs/src/format/table/versioning.md

@@ -35,7 +35,8 @@ they should return an "unsupported" error on any read or write operation.
 | 512      | `FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS` | Yes       | Yes             | The table uses stable row IDs and carries a [Fragment Reuse Index](../index/system/frag_reuse.md). |
 | 1024     | `FLAG_FRAGMENT_REUSE_INDEX`     | Yes             | Yes             | The fragment reuse index records tagged transitions (`IndexMetadata.index_version >= 1`). Readers must translate row addresses through them; writers must preserve them. An implementation without this flag would decode the details as the legacy format and silently drop the transitions when it next rewrites the fragment reuse index. See [FRI index versions](../index/system/frag_reuse.md#fri-index-versions). |
 | 2048     | `FLAG_UNSTABLE_SPILLED_ROW_LINEAGE` | Yes         | Yes             | Some fragment stores its row ids or row version sequences as hidden columns of a data file rather than inline. A reader without this flag would see the fragment as having no row ids. Unstable: release builds reject it unless explicitly opted in. |
+| 4096     | `FLAG_CLUSTERING_METADATA`     | No              | Yes             | The manifest carries [clustering configuration or fragment layout markers](clustering.md). One shared capability, independent of provider and layout version. Reserved, not supported until writer preservation and invalidation are implemented. Readers may ignore the metadata. |
 
 </div>
 
-Flags with bit values 4096 and above are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.
+Flags with bit values 8192 and above are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.

protos/table.proto

@@ -32,6 +32,34 @@ message UUID {
   bytes uuid = 1;
 }
 
+// Identifies the provider and layout version that organized a fragment.
+message FragmentClustering {
+  // Non-empty, case-sensitive implementation name. Namespaced names are recommended.
+  string provider = 1;
+  /* Provider-assigned layout identifier. Zero is invalid. This is neither a
+   * dataset version nor the version of the provider's software.
+   */
+  uint64 version = 2;
+}
+
+// Current table-level clustering configuration. History is provider-managed.
+message Clustering {
+  /* Ordered, non-empty list of distinct schema field IDs, not column names.
+   * These identify all columns on which the layout depends.
+   */
+  repeated int32 columns = 1;
+  // Non-empty, case-sensitive implementation name. Namespaced names are recommended.
+  string provider = 2;
+  // Non-zero, provider-assigned layout/configuration version, not a software version.
+  uint64 version = 3;
+  /* Provider-defined configuration, using the same string key-value shape as
+   * Manifest.config. An empty map means no provider-specific configuration.
+   * Unknown keys and values must be preserved without interpretation.
+   * The provider defines value encoding, schema versioning, and compatibility.
+   */
+  map<string, string> provider_metadata = 4;
+}
+
 // Manifest is a global section shared between all the files.
 message Manifest {
   // All fields of the dataset, including the nested fields.
@@ -146,6 +174,8 @@ message Manifest {
    *
    * The flag identities are the same as for reader_feature_flags, but the values of
    * reader_feature_flags and writer_feature_flags are not required to be identical.
+   * Bit 1 << 12 protects clustering configuration and fragment layout markers.
+   * It is writer-only: readers do not need to interpret clustering metadata.
    */
   uint64 writer_feature_flags = 10;
 
@@ -243,6 +273,11 @@ message Manifest {
 
   // The branch of the dataset. None means main branch.
   optional string branch = 20;
+
+  /* Optional current clustering configuration. Absence disables new clustering
+   * work without clearing existing fragment markers. Requires writer flag 1 << 12.
+   */
+  Clustering clustering = 22;
 } // Manifest
 
 // external dataset base path
@@ -457,6 +492,12 @@ message DataFragment {
    * `deletion_file.num_deleted_rows` from this value.
    */
   uint64 physical_rows = 4;
+
+  /* Optional provider and layout version that organized this fragment.
+   * Absence makes no clustering claim. The version may differ from the current
+   * table configuration; a generic rewrite must not inherit this marker.
+   */
+  FragmentClustering clustering = 15;
 }
 
 message DataFile {

rust/lance-table/benches/manifest_intern.rs

@@ -63,6 +63,7 @@ fn make_uniform_pb_fragments(n: u64, num_fields: usize) -> Vec<pb::DataFragment>
             deletion_file: None,
             row_id_sequence: None,
             physical_rows: 1000,
+            clustering: None,
             last_updated_at_version_sequence: Some(
                 pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions(
                     version_bytes.clone(),
@@ -140,6 +141,7 @@ fn make_diverse_pb_fragments(
                 deletion_file: None,
                 row_id_sequence: None,
                 physical_rows: 1000,
+                clustering: None,
                 last_updated_at_version_sequence: Some(
                     pb::data_fragment::LastUpdatedAtVersionSequence::InlineLastUpdatedAtVersions(
                         version_payloads[version_idx].clone(),

rust/lance-table/src/feature_flags.rs

@@ -80,8 +80,14 @@ pub const FLAG_FRAGMENT_REUSE_INDEX: u64 = 1 << 10;
 /// in debug builds or when [`ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE_ENV`] is set,
 /// mirroring [`FLAG_UNSTABLE_DATA_OVERLAY_FILES`].
 pub const FLAG_UNSTABLE_SPILLED_ROW_LINEAGE: u64 = 1 << 11;
+/// Writer-only protection for provider-neutral clustering metadata.
+///
+/// Reserved, not supported, until writers preserve configuration and maintain
+/// fragment markers across every operation. This is one common capability,
+/// not a bit per provider or layout version. Ordinary readers need no bit.
+pub const FLAG_CLUSTERING_METADATA: u64 = 1 << 12;
 /// The first bit that is unknown as a feature flag
-pub const FLAG_UNKNOWN: u64 = 1 << 12;
+pub const FLAG_UNKNOWN: u64 = 1 << 13;
 
 const _: () = assert!(FLAG_COVERED_INDEX_METADATA < FLAG_UNKNOWN);
 // The fence needs a bit the current released build already refuses, which means
@@ -94,6 +100,7 @@ const _: () = assert!(FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS >= 1 << 8);
 const _: () = assert!(FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS < FLAG_UNKNOWN);
 const _: () = assert!(FLAG_FRAGMENT_REUSE_INDEX < FLAG_UNKNOWN);
 const _: () = assert!(FLAG_UNSTABLE_SPILLED_ROW_LINEAGE < FLAG_UNKNOWN);
+const _: () = assert!(FLAG_CLUSTERING_METADATA < FLAG_UNKNOWN);
 
 pub(crate) const STICKY_PAIRED_FLAGS: u64 =
     FLAG_MIXED_DATA_FILE_VERSIONS | FLAG_FRAGMENT_REUSE_INDEX;
@@ -249,6 +256,7 @@ fn supported_flags_when(overlay_enabled: bool, spilled_row_lineage_enabled: bool
     );
     // Reserved, not implemented: see the flag's doc comment.
     mark_supported(&mut supported, FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS, false);
+    mark_supported(&mut supported, FLAG_CLUSTERING_METADATA, false);
     mark_supported(
         &mut supported,
         FLAG_UNSTABLE_SPILLED_ROW_LINEAGE,
@@ -379,6 +387,21 @@ mod tests {
     use super::*;
     use crate::format::BasePath;
 
+    #[test]
+    fn test_clustering_flag_fences_writes_without_blocking_reads() {
+        let mut manifest = empty_manifest();
+        manifest.writer_feature_flags = FLAG_CLUSTERING_METADATA;
+        ensure_can_read_manifest(&manifest).unwrap();
+        assert!(matches!(
+            ensure_can_write_manifest(&manifest).unwrap_err(),
+            Error::NotSupported { .. }
+        ));
+        assert!(!can_read_dataset(FLAG_CLUSTERING_METADATA));
+        assert!(!can_write_dataset(FLAG_CLUSTERING_METADATA));
+        assert_eq!(FLAG_CLUSTERING_METADATA, 1 << 12);
+        assert_eq!(FLAG_UNKNOWN, 1 << 13);
+    }
+
     /// Reserved ahead of its implementation: refused for reading and writing
     /// until the handling lands, so a build from the gap cannot open the table.
     #[test]

rust/lance-table/src/format/fragment.rs

@@ -778,6 +778,7 @@ impl From<&Fragment> for pb::DataFragment {
             deletion_file,
             row_id_sequence,
             physical_rows: f.physical_rows.unwrap_or_default() as u64,
+            clustering: None,
             last_updated_at_version_sequence,
             created_at_version_sequence,
         }