diff --git a/CHANGELOG.md b/CHANGELOG.md index 617a3d1..89a0115 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,66 @@ All notable changes to ElyraSQL are documented here. The format is based on ## [Unreleased] +### Added + +- **Numeric `RANGE` and `GROUPS` window frames.** Aggregate windows now support + exact numeric offsets in ascending and descending order, peer groups, + partitions, NULL ordering, and empty frames. Integer and decimal boundaries + use checked fixed-point arithmetic rather than lossy floating-point + conversion; invalid, row-dependent, temporal, and incompatible offsets are + rejected explicitly. +- **Composite secondary-index prefix ranges.** Predicates such as + `tenant = ? AND status = ? AND created BETWEEN ? AND ?` can scan the matching + left prefix of a composite index. Bounds honor each component's collation, + repeated constraints are merged, residual predicates are rechecked, and + transactional overlays remain visible. Covered `COUNT(*)` can count index + entries without fetching table rows. +- **`ALTER TABLE ... ADD PRIMARY KEY` for populated tables.** Existing rows and + secondary indexes are reclustered atomically, with serializable range + validation preventing concurrent inserts from surviving in the old row-id + keyspace. +- **Bounded spill-backed `SELECT DISTINCT`.** Large distinct sets now sort and + stream through temporary files instead of requiring the entire result in + memory, while preserving SQL collation, mixed-numeric equality, stable first + representatives, offsets, limits, cancellation, and result metadata. + +### Changed + +- **Correlated `EXISTS` / `NOT EXISTS` can execute as one-time membership + plans.** Safe single-table equality correlations are evaluated once with + exact type/collation gates and correct NULL anti/semi-join semantics; other + shapes retain the general correlated path. `EXPLAIN` reports the optimized + plan only when it is guaranteed. +- **Selective inner joins delay partner-table materialization.** A selective + point driver can probe a partner primary or secondary index directly, + including transaction-local rows, instead of eagerly scanning every joined + table. +- **Window aggregation is incremental where possible.** `SUM`, `COUNT`, and + `AVG` over `RANGE`/`GROUPS` frames use precomputed bounds and prefix state; + `MIN` and `MAX` retain their exact fallback while sharing the faster bound + planning. +- **Bulk inserts and table rewrites do less allocation and redundant work.** + Index keys encode selected columns without cloning, writes are ordered for + the B-tree, unchanged rows reuse their serialized representation, and + serializable scans coalesce overlapping validation ranges. +- **`LOAD DATA INFILE` uses bounded 50,000-row bulk units** to amortize SQL + parsing and durable commits. Insert paths cache trigger definitions (including + empty sets) with DDL-safe invalidation. On the 50,000-row comparison workload, + ordinary 1,000-row batches improved from 1,072 ms to 781 ms, one bulk + statement took 541 ms, and server-side `LOAD DATA` took 267 ms. + +### Fixed + +- Exact `RANGE` boundaries no longer merge distinct integers above `2^53`, and + wholly out-of-partition frames return an empty frame instead of indexing with + `usize::MAX` and crashing the connection. +- External sorting now returns no rows for `LIMIT 0` in every spill/top-N mode + and reports truncated spill headers or bodies as storage corruption rather + than clean EOF or a generic I/O failure. +- Spill-backed `DISTINCT` groups by its canonical SQL key rather than a broader + sort comparison, preventing mixed numeric representations from producing + duplicate output. + ## [1.9.4] - 2026-08-03 Seven contributions, and for the first time in a while none of them is a diff --git a/bench/features_compare.py b/bench/features_compare.py new file mode 100644 index 0000000..129d2e2 --- /dev/null +++ b/bench/features_compare.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +"""Compare recently added SQL paths through one persistent MySQL connection.""" + +import argparse +import math +import statistics +import tempfile +import time +from pathlib import Path + +import pymysql + + +def sample(cur, sql, repeats): + for _ in range(3): + cur.execute(sql) + cur.fetchall() + times = [] + result = None + for _ in range(repeats): + started = time.perf_counter_ns() + cur.execute(sql) + result = cur.fetchall() + times.append((time.perf_counter_ns() - started) / 1_000_000) + ordered = sorted(times) + p95 = ordered[min(len(ordered) - 1, math.ceil(len(ordered) * 0.95) - 1)] + return statistics.median(times), p95, result + + +def batches(cur, table, rows, render, size=1000): + for start in range(0, rows, size): + values = ",".join(render(i) for i in range(start, min(rows, start + size))) + cur.execute(f"INSERT INTO {table} VALUES {values}") + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--port", type=int, required=True) + parser.add_argument("--label", required=True) + parser.add_argument("--database", default="") + parser.add_argument("--rows", type=int, default=50_000) + parser.add_argument("--batch-rows", type=int, default=1_000) + parser.add_argument("--load-data", action="store_true") + args = parser.parse_args() + if args.rows <= 0 or args.batch_rows <= 0: + parser.error("--rows and --batch-rows must be positive") + + conn = pymysql.connect( + host="127.0.0.1", port=args.port, user="root", password="", autocommit=True + ) + cur = conn.cursor() + if args.database: + cur.execute(f"CREATE DATABASE IF NOT EXISTS {args.database}") + cur.execute(f"USE {args.database}") + + for table in ( + "feature_items", + "feature_users", + "feature_orders", + "feature_rekey", + "feature_load", + ): + cur.execute(f"DROP TABLE IF EXISTS {table}") + cur.execute( + "CREATE TABLE feature_items (" + "id BIGINT PRIMARY KEY, tenant BIGINT NOT NULL, created BIGINT NOT NULL, " + "grp BIGINT, val BIGINT, label VARCHAR(32), " + "INDEX tenant_created (tenant, created))" + ) + cur.execute("CREATE TABLE feature_users (id BIGINT PRIMARY KEY, name VARCHAR(32))") + cur.execute("CREATE TABLE feature_orders (id BIGINT PRIMARY KEY, user_id BIGINT)") + + started = time.perf_counter_ns() + batches( + cur, + "feature_items", + args.rows, + lambda i: f"({i},{i % 100},{i},{i % 200},{i % 1000},'label{i % 1000}')", + size=args.batch_rows, + ) + insert_ms = (time.perf_counter_ns() - started) / 1_000_000 + batches( + cur, + "feature_users", + args.rows, + lambda i: f"({i},'user{i}')", + size=args.batch_rows, + ) + batches( + cur, + "feature_orders", + args.rows, + lambda i: f"({i},{(i * 17) % args.rows})", + size=args.batch_rows, + ) + cur.execute("CREATE INDEX orders_user ON feature_orders(user_id)") + for table in ("feature_items", "feature_users", "feature_orders"): + cur.execute(f"ANALYZE TABLE {table}") + cur.fetchall() + + midpoint = args.rows // 2 + workloads = [ + ("PK point lookup", f"SELECT name FROM feature_users WHERE id={midpoint}", 100), + ( + "composite prefix range", + "SELECT COUNT(*) FROM feature_items " + "WHERE tenant=42 AND created BETWEEN 10000 AND 40000", + 40, + ), + ( + "DISTINCT 1000 groups", + "SELECT COUNT(*) FROM (SELECT DISTINCT label FROM feature_items) d", + 15, + ), + ( + "correlated EXISTS", + "SELECT COUNT(*) FROM feature_users u WHERE EXISTS " + "(SELECT 1 FROM feature_orders o WHERE o.user_id=u.id)", + 10, + ), + ( + "selective indexed join", + f"SELECT u.name,o.id FROM feature_users u JOIN feature_orders o " + f"ON u.id=o.user_id WHERE u.id={midpoint}", + 50, + ), + ] + + # Keep the window input bounded: this exposes frame-algorithm scaling without + # letting one quadratic implementation monopolize the benchmark machine. + window_rows = min(args.rows, 5_000) + workloads.append( + ( + f"RANGE window ({window_rows} rows)", + "SELECT SUM(running_sum) FROM (" + "SELECT SUM(val) OVER (ORDER BY created RANGE BETWEEN 10 PRECEDING " + f"AND CURRENT ROW) running_sum FROM feature_items WHERE id < {window_rows}) w", + 5, + ) + ) + + results = [] + for name, sql, repeats in workloads: + median, p95, result = sample(cur, sql, repeats) + results.append((name, median, p95, result)) + + rekey_rows = min(args.rows, 20_000) + cur.execute("CREATE TABLE feature_rekey (id BIGINT, payload VARCHAR(32))") + batches( + cur, + "feature_rekey", + rekey_rows, + lambda i: f"({i},'row{i}')", + size=args.batch_rows, + ) + started = time.perf_counter_ns() + cur.execute("ALTER TABLE feature_rekey ADD PRIMARY KEY (id)") + rekey_ms = (time.perf_counter_ns() - started) / 1_000_000 + + load_ms = None + load_error = None + if args.load_data: + cur.execute( + "CREATE TABLE feature_load (id BIGINT PRIMARY KEY, payload VARCHAR(32))" + ) + with tempfile.NamedTemporaryFile( + mode="w", prefix="elyra-load-", suffix=".tsv", delete=False + ) as load_file: + load_path = Path(load_file.name) + for i in range(args.rows): + load_file.write(f"{i}\trow{i}\n") + try: + started = time.perf_counter_ns() + try: + cur.execute( + f"LOAD DATA INFILE '{load_path}' INTO TABLE feature_load " + "FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'" + ) + load_ms = (time.perf_counter_ns() - started) / 1_000_000 + cur.execute("SELECT COUNT(*) FROM feature_load") + loaded = cur.fetchone()[0] + if loaded != args.rows: + raise RuntimeError(f"LOAD DATA stored {loaded} of {args.rows} rows") + except pymysql.MySQLError as error: + load_error = str(error) + finally: + load_path.unlink(missing_ok=True) + + print(f"\n{args.label}: {args.rows:,} rows") + print(f"{'workload':<34} {'median ms':>12} {'p95 ms':>12}") + print("-" * 60) + print(f"{'bulk insert feature_items':<34} {insert_ms:>12.2f} {insert_ms:>12.2f}") + for name, median, p95, _ in results: + print(f"{name:<34} {median:>12.2f} {p95:>12.2f}") + print(f"{f'ADD PRIMARY KEY ({rekey_rows:,})':<34} {rekey_ms:>12.2f} {rekey_ms:>12.2f}") + if load_ms is not None: + print(f"{f'LOAD DATA ({args.rows:,})':<34} {load_ms:>12.2f} {load_ms:>12.2f}") + elif load_error is not None: + print(f"{'LOAD DATA':<34} {'unavailable':>12} {'unavailable':>12}") + print(f" {load_error}") + + cur.close() + conn.close() + + +if __name__ == "__main__": + main() diff --git a/crates/elyra-engine/src/catalog.rs b/crates/elyra-engine/src/catalog.rs index e6c8e14..aa06bf8 100644 --- a/crates/elyra-engine/src/catalog.rs +++ b/crates/elyra-engine/src/catalog.rs @@ -306,12 +306,39 @@ pub fn trigname_key(name: &str) -> Vec { /// Load all triggers defined on `table`. pub async fn load_triggers(db: &Session, table: &str) -> Result> { + let epoch = CATALOG_EPOCH.load(Ordering::Acquire); + let cache_key = (db.db_id(), table.to_ascii_lowercase()); + if !db.in_txn() { + if let Some((cached_epoch, triggers)) = trigger_cache().read().unwrap().get(&cache_key) { + if *cached_epoch == epoch { + return Ok((**triggers).clone()); + } + } + } let prefix = trigger_prefix(table); let batch = db.scan_batch(prefix, None, 4096).await?; - Ok(batch + let triggers: Vec = batch .iter() .filter_map(|(_, v)| bincode::deserialize(v).ok()) - .collect()) + .collect(); + if !db.in_txn() { + trigger_cache() + .write() + .unwrap() + .insert(cache_key, (epoch, std::sync::Arc::new(triggers.clone()))); + } + Ok(triggers) +} + +#[allow(clippy::type_complexity)] +fn trigger_cache() -> &'static std::sync::RwLock< + std::collections::HashMap<(u64, String), (u64, std::sync::Arc>)>, +> { + use std::sync::{OnceLock, RwLock}; + static CACHE: OnceLock< + RwLock>)>>, + > = OnceLock::new(); + CACHE.get_or_init(|| RwLock::new(std::collections::HashMap::new())) } /// Find a trigger by name (for DROP TRIGGER) via the name->table index — O(1), diff --git a/crates/elyra-engine/src/exec.rs b/crates/elyra-engine/src/exec.rs index 678c0c2..bc8c77f 100644 --- a/crates/elyra-engine/src/exec.rs +++ b/crates/elyra-engine/src/exec.rs @@ -855,8 +855,168 @@ fn explain_first_table(stmt: &sqlparser::ast::Statement) -> Option { None } -/// `EXPLAIN ` — a best-effort, MySQL-shaped plan row (names the first -/// base table and its estimated row count). Not a full optimizer trace. +struct ExplainAccess { + kind: &'static str, + possible_keys: Option, + key: Option, + rows: String, + extra: String, +} + +#[derive(Default)] +struct ExplainFeatureVisitor { + incremental_window: bool, +} + +impl Visitor for ExplainFeatureVisitor { + type Break = (); + + fn pre_visit_expr(&mut self, expression: &Expr) -> ControlFlow { + if let Expr::Function(function) = expression { + self.incremental_window |= function.over.is_some() + && window_aggregate_is_incremental(&function_name(function)); + } + ControlFlow::Continue(()) + } +} + +async fn explain_first_access( + db: &Session, + stmt: &sqlparser::ast::Statement, + table: &str, + rows_estimate: String, +) -> Result { + use sqlparser::ast::{SetExpr, Statement}; + let select = match stmt { + Statement::Query(query) => match query.body.as_ref() { + SetExpr::Select(select) => Some(select.as_ref()), + _ => None, + }, + _ => None, + }; + let selection = select.and_then(|select| select.selection.as_ref()); + let def = match catalog::load(db, table).await { + Ok(def) => def, + Err(Error::Catalog(_)) => { + return Ok(ExplainAccess { + kind: "ALL", + possible_keys: None, + key: None, + rows: rows_estimate, + extra: selection.map_or_else(String::new, |_| "Using where".into()), + }); + } + Err(error) => return Err(error), + }; + let mut feature_extra = Vec::new(); + if let (Some(filter), Some(from)) = (selection, select.and_then(|select| select.from.first())) { + let outer = factor_qualifier_object(db, &from.relation) + .map(|qualifier| object_name_parts(&qualifier)) + .unwrap_or_else(|| vec![table.to_string()]); + if correlated_exists_membership_eligible(db, filter, &def, &outer).await? { + feature_extra.push("Using semi-join membership"); + } + } + if select.is_some_and(|select| select.distinct.is_some()) { + feature_extra.push("Distinct (spill-capable)"); + } + if let Some(select) = select { + let mut visitor = ExplainFeatureVisitor::default(); + let _ = select.visit(&mut visitor); + if visitor.incremental_window { + feature_extra.push("Incremental window aggregate"); + } + } + let decorate = |mut access: ExplainAccess| { + if !feature_extra.is_empty() { + if !access.extra.is_empty() { + access.extra.push_str("; "); + } + access.extra.push_str(&feature_extra.join("; ")); + } + access + }; + if def.has_pk() && key_eq_values(&def, selection, &def.pk_cols)?.is_some() { + return Ok(decorate(ExplainAccess { + kind: "const", + possible_keys: Some("PRIMARY".into()), + key: Some("PRIMARY".into()), + rows: "1".into(), + extra: "Using where".into(), + })); + } + for index in &def.indexes { + if !index.vector && key_eq_values(&def, selection, &index.cols)?.is_some() { + return Ok(decorate(ExplainAccess { + kind: "ref", + possible_keys: Some(index.name.clone()), + key: Some(index.name.clone()), + rows: "1".into(), + extra: "Using index condition; Using where".into(), + })); + } + } + if let Some(range) = composite_range_bounds(&def, selection)? { + return Ok(decorate(ExplainAccess { + kind: "range", + possible_keys: Some(range.index.name.clone()), + key: Some(range.index.name.clone()), + rows: rows_estimate, + extra: "Using index condition; Using where".into(), + })); + } + if let Some(range) = range_bounds(&def, selection)? { + let key = if def.pk_cols == [range.col] { + "PRIMARY".to_string() + } else { + index::index_on(&def, range.col) + .map(|index| index.name.clone()) + .unwrap_or_default() + }; + return Ok(decorate(ExplainAccess { + kind: "range", + possible_keys: Some(key.clone()), + key: Some(key), + rows: rows_estimate, + extra: "Using index condition; Using where".into(), + })); + } + Ok(decorate(ExplainAccess { + kind: "ALL", + possible_keys: None, + key: None, + rows: rows_estimate, + extra: selection.map_or_else(String::new, |_| "Using where".into()), + })) +} + +fn explain_row(table: Option, access: ExplainAccess, extra: Option<&str>) -> Vec { + let mut access_extra = access.extra; + if let Some(extra) = extra { + if !access_extra.is_empty() { + access_extra.push_str("; "); + } + access_extra.push_str(extra); + } + vec![ + Value::Text("1".into()), + Value::Text("SIMPLE".into()), + table.map(Value::Text).unwrap_or(Value::Null), + Value::Null, + Value::Text(access.kind.into()), + access.possible_keys.map(Value::Text).unwrap_or(Value::Null), + access.key.map(Value::Text).unwrap_or(Value::Null), + Value::Null, + Value::Null, + Value::Text(access.rows), + Value::Text("100.00".into()), + Value::Text(access_extra), + ] +} + +/// `EXPLAIN ` — a MySQL-shaped summary of access paths that the +/// executor can prove it will use. It remains a compact trace rather than a +/// full cost model. pub async fn explain(db: &Session, stmt: &sqlparser::ast::Statement) -> Result { let schema = text_schema(&[ "id", @@ -872,7 +1032,25 @@ pub async fn explain(db: &Session, stmt: &sqlparser::ast::Statement) -> Result match query.body.as_ref() { + SetExpr::Select(select) => Some(select.as_ref()), + _ => None, + }, + _ => None, + }; + let indexed_join = match select { + Some(select) => match guaranteed_indexed_join_access(db, select).await { + Ok(access) => access, + Err(Error::Catalog(_) | Error::UnknownDatabase(_)) => None, + Err(error) => return Err(error), + }, + None => None, + }; + let table = indexed_join + .as_ref() + .map(|join| join.driver_table.clone()) + .or_else(|| explain_first_table(stmt)); let rows_est = match &table { Some(t) => catalog::load_stats(db, t) .await? @@ -880,21 +1058,35 @@ pub async fn explain(db: &Session, stmt: &sqlparser::ast::Statement) -> Result "0".into(), }; - let row = vec![ - Value::Text("1".into()), - Value::Text("SIMPLE".into()), - table.clone().map(Value::Text).unwrap_or(Value::Null), - Value::Null, - Value::Text(if table.is_some() { "ALL" } else { "" }.into()), - Value::Null, - Value::Null, - Value::Null, - Value::Null, - Value::Text(rows_est), - Value::Text("100.00".into()), - Value::Text(String::new()), - ]; - Ok(QueryResult::Rows(RowStream::literal(schema, vec![row]))) + let access = match table.as_deref() { + Some(table) => explain_first_access(db, stmt, table, rows_est).await?, + None => ExplainAccess { + kind: "", + possible_keys: None, + key: None, + rows: rows_est, + extra: String::new(), + }, + }; + let mut rows = vec![explain_row(table, access, None)]; + if let Some(join) = indexed_join { + let partner_rows = catalog::load_stats(db, &join.partner_table) + .await? + .map(|stats| stats.rows.to_string()) + .unwrap_or_else(|| "0".into()); + rows.push(explain_row( + Some(join.partner_table), + ExplainAccess { + kind: join.access_type, + possible_keys: Some(join.index_name.clone()), + key: Some(join.index_name), + rows: partner_rows, + extra: "Using index condition".into(), + }, + Some("Indexed nested-loop join"), + )); + } + Ok(QueryResult::Rows(RowStream::literal(schema, rows))) } /// MySQL-compatible system variables reported by `SHOW VARIABLES`. ElyraSQL @@ -2694,23 +2886,24 @@ pub async fn alter_table( if implicit_transaction { db.begin()?; } - let checkpoint = match db.transaction_checkpoint() { - Ok(checkpoint) => checkpoint, - Err(error) => { - if implicit_transaction { - db.rollback(); - } - return Err(error); - } + // ALTER helpers that rewrite a table scan a snapshot before staging new + // keys. Validate those scanned ranges at commit so a concurrent write + // cannot survive in an obsolete row/key layout. + db.require_serializable_validation()?; + // An implicit ALTER owns its whole transaction, so an error can discard it + // directly. Checkpoint logging would clone every rewritten key solely to + // support a partial rollback that can never be needed. Explicit user + // transactions still need a checkpoint to preserve earlier statements. + let checkpoint = if implicit_transaction { + None + } else { + Some(db.transaction_checkpoint()?) }; match alter_table_inner(db, name, ops).await { Ok(result) => { - if let Err(error) = db.release_transaction_checkpoint(checkpoint) { - if implicit_transaction { - db.rollback(); - } - return Err(error); + if let Some(checkpoint) = checkpoint { + db.release_transaction_checkpoint(checkpoint)?; } if implicit_transaction { db.commit().await?; @@ -2718,11 +2911,8 @@ pub async fn alter_table( Ok(result) } Err(error) => { - if let Err(rollback_error) = db.rollback_transaction_checkpoint(checkpoint) { - if implicit_transaction { - db.rollback(); - } - return Err(rollback_error); + if let Some(checkpoint) = checkpoint { + db.rollback_transaction_checkpoint(checkpoint)?; } if implicit_transaction { db.rollback(); @@ -2885,30 +3075,28 @@ async fn alter_table_inner( }); continue; } - let (idx_name, columns, unique) = - match tc { - TC::Index { name, columns, .. } => (name.clone(), columns.clone(), false), - TC::Unique { - name, - index_name, - columns, - .. - } => ( - name.clone().or_else(|| index_name.clone()), - columns.clone(), - true, - ), - TC::PrimaryKey { .. } => return Err(Error::Unsupported( - "ALTER TABLE ADD PRIMARY KEY on an existing table is not supported; \ - declare the primary key in CREATE TABLE" - .into(), - )), - other => { - return Err(Error::Unsupported(format!( - "ALTER ADD constraint not supported: {other}" - ))) - } - }; + let (idx_name, columns, unique) = match tc { + TC::Index { name, columns, .. } => (name.clone(), columns.clone(), false), + TC::Unique { + name, + index_name, + columns, + .. + } => ( + name.clone().or_else(|| index_name.clone()), + columns.clone(), + true, + ), + TC::PrimaryKey { columns, .. } => { + alter_add_primary_key(db, &mut def, columns).await?; + continue; + } + other => { + return Err(Error::Unsupported(format!( + "ALTER ADD constraint not supported: {other}" + ))) + } + }; let ci = CreateIndex { name: idx_name.map(|i| ObjectName(vec![i])), table_name: name.clone(), @@ -2948,6 +3136,122 @@ async fn alter_table_inner( Ok(QueryResult::Affected(0)) } +/// Add a clustered primary key to a rowid table, re-keying every stored row and +/// rebuilding secondary-index entries against the new clustered keys. +async fn alter_add_primary_key(db: &Session, def: &mut TableDef, columns: &[Ident]) -> Result<()> { + if def.has_pk() { + return Err(Error::Query("multiple primary keys are not allowed".into())); + } + if columns.is_empty() { + return Err(Error::Query( + "ALTER TABLE ADD PRIMARY KEY requires at least one column".into(), + )); + } + + let mut pk_cols = Vec::with_capacity(columns.len()); + for column in columns { + let index = def + .schema + .columns + .iter() + .position(|candidate| predicate::identifier_eq(&candidate.name, &column.value)) + .ok_or_else(|| Error::Catalog(format!("unknown column: {column}")))?; + if pk_cols.contains(&index) { + return Err(Error::Query(format!( + "duplicate column '{}' in primary key", + column.value + ))); + } + pk_cols.push(index); + } + + let old_def = def.clone(); + def.pk_cols = pk_cols; + for &column in &def.pk_cols { + def.schema.columns[column].nullable = false; + } + + let mut puts = vec![(catalog_key(&def.name), def.encode()?)]; + let mut deletes = Vec::new(); + let mut clustered_keys = std::collections::HashSet::new(); + let pk_collations = def.pk_collations(); + let clustered_prefix = data_prefix(&def.name); + let rewrite_budget = db.transaction_write_budget_remaining(); + let mut rewrite_bytes = puts + .iter() + .map(|(key, value)| key.len() + value.len()) + .sum(); + let prefix = data_prefix(&old_def.name); + let mut cursor = None; + loop { + let batch = db.scan_batch(prefix.clone(), cursor.clone(), 4096).await?; + if batch.is_empty() { + break; + } + let last = batch.len() < 4096; + cursor = batch.last().map(|(key, _)| key.clone()); + for (old_key, encoded_row) in batch { + let row = rowdec::decode_row(&encoded_row)?; + if def.pk_cols.iter().any(|&column| row[column].is_null()) { + return Err(Error::Query( + "primary key columns cannot contain NULL".into(), + )); + } + let clustered = keyenc::encode_columns_coll(&row, &def.pk_cols, &pk_collations)?; + if !clustered_keys.insert(clustered.clone()) { + return Err(Error::Duplicate("duplicate primary key".into())); + } + let mut new_key = clustered_prefix.clone(); + new_key.extend_from_slice(&clustered); + let mut row_deletes = index::entry_keys_for_row(&old_def, &row, &old_key)?; + if new_key != old_key { + row_deletes.push(old_key); + } + let mut row_puts = vec![(new_key.clone(), encoded_row)]; + row_puts.extend(index::entries_for_row(def, &row, &new_key)?); + let additional = clustered.len() + + row_deletes.iter().map(Vec::len).sum::() + + row_puts + .iter() + .map(|(key, value)| key.len() + value.len()) + .sum::(); + rewrite_bytes = reserve_alter_rewrite_bytes(rewrite_bytes, additional, rewrite_budget)?; + deletes.extend(row_deletes); + puts.extend(row_puts); + } + if last { + break; + } + } + + deletes.push(rowid_key(&def.name)); + puts.push(bump_wcount(db, &def.name).await?); + db.commit_write(puts, deletes).await +} + +fn reserve_alter_rewrite_bytes(current: usize, additional: usize, budget: usize) -> Result { + let total = current.saturating_add(additional); + if total > budget { + return Err(Error::Query(format!( + "ALTER TABLE rewrite exceeded {budget} bytes; raise \ + ELYRASQL_TXN_MAX_BYTES to allow a larger rewrite" + ))); + } + Ok(total) +} + +#[cfg(test)] +mod alter_rewrite_budget_tests { + use super::reserve_alter_rewrite_bytes; + + #[test] + fn rejects_before_the_rewrite_buffer_exceeds_its_budget() { + assert_eq!(reserve_alter_rewrite_bytes(60, 40, 100).unwrap(), 100); + let error = reserve_alter_rewrite_bytes(60, 41, 100).unwrap_err(); + assert!(error.to_string().contains("rewrite exceeded 100 bytes")); + } +} + fn ensure_col_meta(def: &mut TableDef) { if def.col_meta.len() < def.schema.columns.len() { def.col_meta @@ -4016,6 +4320,7 @@ pub async fn insert(db: &Session, vindex: &VectorRegistry, ins: Insert) -> Resul let on_dup = !dup_sets.is_empty(); let has_pk = def.has_pk(); let pk_colls = def.pk_collations(); + let clustered_prefix = data_prefix(&name); // Load rowid counter once for tables without a PK. let mut next_rowid = if has_pk { @@ -4179,11 +4484,15 @@ pub async fn insert(db: &Session, vindex: &VectorRegistry, ins: Insert) -> Resul check_row(&def, &checks, &row)?; let key = if has_pk { - let pk_vals: Vec = def.pk_cols.iter().map(|&i| row[i].clone()).collect(); - data_key(&name, &keyenc::encode_key_coll(&pk_vals, &pk_colls)?) + let encoded = keyenc::encode_columns_coll(&row, &def.pk_cols, &pk_colls)?; + let mut key = clustered_prefix.clone(); + key.extend_from_slice(&encoded); + key } else { next_rowid += 1; - data_key(&name, &keyenc::encode_rowid(next_rowid)) + let mut key = clustered_prefix.clone(); + key.extend_from_slice(&keyenc::encode_rowid(next_rowid)); + key }; built.push((key, row)); } @@ -4219,6 +4528,11 @@ pub async fn insert(db: &Session, vindex: &VectorRegistry, ins: Insert) -> Resul if auto_col.is_some() { aux_puts.push((autoinc_key(&name), autoinc.to_le_bytes().to_vec())); } + // redb's B-tree writer benefits materially from monotonic key order. + // These sets have order-independent semantics: every `new` key must be + // unique and auxiliary index/counter keys contain no competing writes. + new_puts.sort_unstable_by(|left, right| left.0.cmp(&right.0)); + aux_puts.sort_unstable_by(|left, right| left.0.cmp(&right.0)); let affected = built.len() as u64; db.raw_db() .commit_insert(new_puts, aux_puts, Vec::new()) @@ -5242,34 +5556,12 @@ async fn select_inner( si.distinct = None; } let res = Box::pin(select(db, vindex, &inner_q)).await?; - let QueryResult::Rows(mut stream) = res else { + let QueryResult::Rows(stream) = res else { return Ok(res); }; - let schema = stream.schema.clone(); - let colls: Vec = - schema.columns.iter().map(|c| c.collation).collect(); - let mut seen: std::collections::HashSet> = std::collections::HashSet::new(); - let mut out: Vec> = Vec::new(); - let cap = distinct_max(); - loop { - let batch = stream.next_batch(8192).await?; - if batch.is_empty() { - break; - } - for row in batch { - if seen.insert(Value::row_collation_key_coll(&row, &colls)) { - out.push(row); - if out.len() > cap { - return Err(Error::Query(format!( - "SELECT DISTINCT exceeded {cap} distinct rows; narrow the query \ - or raise ELYRASQL_DISTINCT_MAX" - ))); - } - } - } - } - apply_offset_limit(&mut out, d_offset, d_limit); - return Ok(QueryResult::Rows(RowStream::literal(schema, out))); + return Ok(QueryResult::Rows( + distinct_rows(stream, d_offset, d_limit, distinct_max(), db.cancel_token()).await?, + )); } } @@ -8772,33 +9064,153 @@ struct RangeQuery { hi: Option<(Value, bool)>, } -/// Detect a range over a PK/indexed column from the filter's AND-conjuncts -/// (`col >|>=|<|<= lit`, `col BETWEEN a AND b`). Only columns with -/// order-encodable bound values qualify. -fn range_bounds(def: &TableDef, filter: Option<&Expr>) -> Result> { - use std::collections::HashMap; - let Some(f) = filter else { return Ok(None) }; - let mut conj = Vec::new(); - split_and(f, &mut conj); +/// An equality-constrained leading prefix followed by a range on the next +/// column of a composite secondary index. +struct CompositeRangeQuery<'a> { + index: &'a IndexDef, + prefix: Vec, + lo: Option<(Value, bool)>, + hi: Option<(Value, bool)>, +} - type Bounds = (Option<(Value, bool)>, Option<(Value, bool)>); - let mut map: HashMap = HashMap::new(); +type RangeBound = Option<(Value, bool)>; +type ColumnBounds = (RangeBound, RangeBound); +type EqualityConstraints = std::collections::HashMap; +type RangeConstraints = std::collections::HashMap; - for c in &conj { - if let Some((col, op, val)) = as_range(def, c)? { - let e = map.entry(col).or_default(); - use sqlparser::ast::BinaryOperator::*; +fn merge_lower_bound( + current: &mut Option<(Value, bool)>, + candidate: (Value, bool), + collation: elyra_core::Collation, +) { + let replace = match current { + None => true, + Some((value, inclusive)) => match candidate.0.compare_coll(value, collation) { + Some(std::cmp::Ordering::Greater) => true, + Some(std::cmp::Ordering::Equal) => *inclusive && !candidate.1, + _ => false, + }, + }; + if replace { + *current = Some(candidate); + } +} + +fn merge_upper_bound( + current: &mut Option<(Value, bool)>, + candidate: (Value, bool), + collation: elyra_core::Collation, +) { + let replace = match current { + None => true, + Some((value, inclusive)) => match candidate.0.compare_coll(value, collation) { + Some(std::cmp::Ordering::Less) => true, + Some(std::cmp::Ordering::Equal) => *inclusive && !candidate.1, + _ => false, + }, + }; + if replace { + *current = Some(candidate); + } +} + +fn predicate_constraints( + def: &TableDef, + filter: Option<&Expr>, +) -> Result<(EqualityConstraints, RangeConstraints)> { + use sqlparser::ast::BinaryOperator::*; + use std::collections::HashMap; + let mut equalities = HashMap::new(); + let mut ranges: RangeConstraints = HashMap::new(); + let Some(filter) = filter else { + return Ok((equalities, ranges)); + }; + let mut conjuncts = Vec::new(); + split_and(filter, &mut conjuncts); + for conjunct in &conjuncts { + if let Some((col, value)) = eq_col_literal(def, Some(conjunct))? { + equalities.entry(col).or_insert(value); + } + if let Some((col, op, value)) = as_range(def, conjunct)? { + let bounds = ranges.entry(col).or_default(); + let collation = def.collation_of(col); match op { - Gt => e.0 = Some((val, false)), - GtEq => e.0 = Some((val, true)), - Lt => e.1 = Some((val, false)), - LtEq => e.1 = Some((val, true)), + Gt => merge_lower_bound(&mut bounds.0, (value, false), collation), + GtEq => merge_lower_bound(&mut bounds.0, (value, true), collation), + Lt => merge_upper_bound(&mut bounds.1, (value, false), collation), + LtEq => merge_upper_bound(&mut bounds.1, (value, true), collation), _ => {} } - } else if let Some((col, lo, hi)) = as_between(def, c)? { - map.insert(col, (Some((lo, true)), Some((hi, true)))); + } else if let Some((col, lo, hi)) = as_between(def, conjunct)? { + let bounds = ranges.entry(col).or_default(); + let collation = def.collation_of(col); + merge_lower_bound(&mut bounds.0, (lo, true), collation); + merge_upper_bound(&mut bounds.1, (hi, true), collation); + } + } + Ok((equalities, ranges)) +} + +fn composite_range_bounds<'a>( + def: &'a TableDef, + filter: Option<&Expr>, +) -> Result>> { + let (equalities, ranges) = predicate_constraints(def, filter)?; + for index in &def.indexes { + if index.vector || index.fulltext || index.cols.len() < 2 { + continue; + } + let prefix_len = index + .cols + .iter() + .take_while(|col| equalities.contains_key(col)) + .count(); + if prefix_len == 0 || prefix_len >= index.cols.len() { + continue; + } + let range_col = index.cols[prefix_len]; + let Some((lo, hi)) = ranges.get(&range_col) else { + continue; + }; + // Composite entries are omitted when *any* indexed component is NULL. + // The equality and range predicates themselves reject NULL in their + // columns, but a nullable trailing component could omit an otherwise + // qualifying row and make this scan incomplete. + if index.cols[prefix_len + 1..] + .iter() + .any(|&col| def.schema.columns[col].nullable) + { + continue; + } + let prefix = index.cols[..prefix_len] + .iter() + .map(|col| equalities[col].clone()) + .collect::>(); + if keyenc::encode_key_coll(&prefix, &index.col_collations).is_err() + || lo.as_ref().is_some_and(|(value, _)| { + keyenc::encode_coll(value, def.collation_of(range_col)).is_err() + }) + || hi.as_ref().is_some_and(|(value, _)| { + keyenc::encode_coll(value, def.collation_of(range_col)).is_err() + }) + { + continue; } + return Ok(Some(CompositeRangeQuery { + index, + prefix, + lo: lo.clone(), + hi: hi.clone(), + })); } + Ok(None) +} + +/// Detect a range over a PK/indexed column from the filter's AND-conjuncts +/// (`col >|>=|<|<= lit`, `col BETWEEN a AND b`). Only columns with +/// order-encodable bound values qualify. +fn range_bounds(def: &TableDef, filter: Option<&Expr>) -> Result> { + let (_, map) = predicate_constraints(def, filter)?; for (col, (lo, hi)) in map { if lo.is_none() && hi.is_none() { @@ -9123,6 +9535,35 @@ async fn index_range( Ok(Some(out)) } +async fn composite_index_range( + db: &Session, + def: &TableDef, + query: &CompositeRangeQuery<'_>, + budget: Option, +) -> Result, Vec)>>> { + let lo = query + .lo + .as_ref() + .map(|(value, inclusive)| (value, *inclusive)); + let hi = query + .hi + .as_ref() + .map(|(value, inclusive)| (value, *inclusive)); + let data_keys = + index::lookup_prefix_range(db, &def.name, query.index, &query.prefix, lo, hi).await?; + if budget.is_some_and(|budget| data_keys.len() > budget) { + return Ok(None); + } + let blobs = db.multi_get(data_keys.clone()).await?; + let mut out = Vec::with_capacity(data_keys.len()); + for (key, blob) in data_keys.into_iter().zip(blobs) { + if let Some(blob) = blob { + out.push((key, rowdec::decode_row(&blob)?)); + } + } + Ok(Some(out)) +} + /// Build the joined row set from a FROM clause (comma cross-joins + explicit /// JOINs), pushing single-table `conjuncts` down to each base relation. async fn build_from( @@ -9673,6 +10114,7 @@ pub fn parse_load_data(sql: &str) -> Result { /// Turn file `content` into batched `INSERT` statements per the load spec. pub fn build_load_inserts(spec: &LoadSpec, content: &str, batch: usize) -> Vec { + let batch = batch.max(1); let mut stmts = Vec::new(); let col_list = if spec.cols.is_empty() { String::new() @@ -9692,7 +10134,7 @@ pub fn build_load_inserts(spec: &LoadSpec, content: &str, batch: usize) -> Vec = Vec::with_capacity(batch); + let mut tuples: Vec = Vec::with_capacity(batch.min(50_000)); for line in rows_iter.by_ref().take(batch) { let fields = line.split(spec.field_term.as_str()).map(|f| { let f = match spec.enclosed { @@ -9719,14 +10161,163 @@ pub fn build_load_inserts(spec: &LoadSpec, content: &str, batch: usize) -> Vec LoadSpec { + LoadSpec { + path: String::new(), + table: "items".into(), + cols: vec!["id".into(), "label".into()], + field_term: "\t".into(), + enclosed: None, + line_term: "\n".into(), + ignore: 0, + } + } + + #[test] + fn load_builder_honors_bulk_boundaries_and_zero_batch() { + let content = "1\tone\n2\ttwo\n3\tthree\n"; + let statements = build_load_inserts(&spec(), content, 2); + assert_eq!(statements.len(), 2); + assert!(statements[0].contains("(\'1\', \'one\'), (\'2\', \'two\')")); + assert!(statements[1].contains("(\'3\', \'three\')")); + + let zero_batch = build_load_inserts(&spec(), content, 0); + assert_eq!(zero_batch.len(), 3); + } +} + /// Execute an all-INNER join chain over base tables in a cost-based order. /// Loads each table (with predicate pushdown), then greedily joins starting from /// the smallest, always extending along an available equi-join predicate. -async fn build_inner_join_reordered( - db: &Session, - vindex: &VectorRegistry, - twj: &TableWithJoins, - conjuncts: &[Expr], +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct GuaranteedIndexedJoinAccess { + pub driver_table: String, + pub partner_table: String, + pub index_name: String, + pub access_type: &'static str, +} + +async fn stored_base_table_name(db: &Session, factor: &TableFactor) -> Result> { + let TableFactor::Table { name, .. } = factor else { + return Ok(None); + }; + let table = match stored_table_ident(db, name) { + Ok(table) => table, + Err(Error::Catalog(_) | Error::UnknownDatabase(_)) => return Ok(None), + Err(error) => return Err(error), + }; + Ok(catalog::exists(db, &table).await?.then_some(table)) +} + +/// Report the subset of selective joins that is guaranteed to use delayed +/// indexed probes: a two-table INNER equi-join with exactly one indexable local +/// predicate, that predicate being a point lookup on the driver's single-column +/// primary key. The PK lookup guarantees at most one driving row, so execution +/// cannot cross [`NLJ_MAX_DRIVING`] and fall back to materialising the partner. +/// +/// This is deliberately narrower than the optimizer. It is suitable for +/// truthful EXPLAIN metadata without reading table rows or changing session +/// state; plans outside the guaranteed subset simply return `None`. +pub(crate) async fn guaranteed_indexed_join_access( + db: &Session, + select: &Select, +) -> Result> { + if select.from.len() != 1 { + return Ok(None); + } + let twj = &select.from[0]; + if twj.joins.len() != 1 + || !stored_table_factor(&twj.relation) + || !stored_table_factor(&twj.joins[0].relation) + { + return Ok(None); + } + if stored_base_table_name(db, &twj.relation).await?.is_none() + || stored_base_table_name(db, &twj.joins[0].relation) + .await? + .is_none() + { + return Ok(None); + } + let (kind, on) = join_kind(&twj.joins[0].join_operator)?; + if kind != JoinKind::Inner { + return Ok(None); + } + let Some(on) = on else { return Ok(None) }; + if !matches!( + on, + Expr::BinaryOp { + op: sqlparser::ast::BinaryOperator::Eq, + .. + } + ) { + return Ok(None); + } + + let (left_def, left_cols) = resolve_table(db, &twj.relation).await?; + let (right_def, right_cols) = resolve_table(db, &twj.joins[0].relation).await?; + let mut conjuncts = Vec::new(); + if let Some(filter) = &select.selection { + split_and(filter, &mut conjuncts); + } + let left_schema = Schema::new(left_cols.clone()); + let right_schema = Schema::new(right_cols.clone()); + let pk_point = |def: &TableDef, schema: &Schema| -> Result { + let mut found = false; + for conjunct in &conjuncts { + if !refs_in_schema(conjunct, schema) { + continue; + } + if let Some((column, _)) = eq_col_literal(def, Some(conjunct))? { + if def.pk_cols == [column] { + found = true; + } else if index::index_on(def, column).is_some() { + // Both sides would be optimizer candidates; without reading + // rows we cannot guarantee which one becomes the driver. + return Ok(false); + } + } + } + Ok(found) + }; + let left_point = pk_point(&left_def, &left_schema)?; + let right_point = pk_point(&right_def, &right_schema)?; + let (driver_schema, partner_schema, driver_table, partner_def) = match (left_point, right_point) + { + (true, false) => (left_schema, right_schema, left_def.name.clone(), right_def), + (false, true) => (right_schema, left_schema, right_def.name.clone(), left_def), + _ => return Ok(None), + }; + let Some((_, partner_col)) = equi_nlj(&on, &driver_schema, &partner_schema) else { + return Ok(None); + }; + if partner_def.pk_cols == [partner_col] { + return Ok(Some(GuaranteedIndexedJoinAccess { + driver_table, + partner_table: partner_def.name, + index_name: "PRIMARY".into(), + access_type: "eq_ref", + })); + } + Ok( + index::index_on(&partner_def, partner_col).map(|idx| GuaranteedIndexedJoinAccess { + driver_table, + partner_table: partner_def.name.clone(), + index_name: idx.name.clone(), + access_type: "ref", + }), + ) +} + +async fn build_inner_join_reordered( + db: &Session, + vindex: &VectorRegistry, + twj: &TableWithJoins, + conjuncts: &[Expr], ) -> Result, Vec>)>> { // Collect relations and ON predicates. A CROSS JOIN contributes no predicate. let mut relations: Vec<&TableFactor> = vec![&twj.relation]; @@ -9755,28 +10346,51 @@ async fn build_inner_join_reordered( if on_preds.len() + 1 < relations.len() { return Ok(None); } + for relation in &relations { + if stored_base_table_name(db, relation).await?.is_none() { + return Ok(None); + } + } - // Load each relation (materialize + pushdown) and estimate its size. - struct Loaded { + // Resolve and estimate each relation without reading its rows. In particular, + // do not eagerly scan a large future partner: once a selective relation has + // become the driver we may be able to probe that partner's index directly. + struct Candidate<'a> { + relation: &'a TableFactor, + def: TableDef, cols: Vec, - rows: Vec>, est: u64, + accelerable: bool, } - let mut loaded: Vec = Vec::with_capacity(relations.len()); + let mut candidates: Vec> = Vec::with_capacity(relations.len()); for rel in &relations { - let (cols, mut rows) = load_relation(db, vindex, rel, conjuncts).await?; - rows = apply_pushdown(rows, &cols, conjuncts)?; - // Prefer the actual loaded size (already filtered) as the cost estimate. - let est = rows.len() as u64; - loaded.push(Loaded { cols, rows, est }); + let (def, cols) = resolve_table(db, rel).await?; + let schema = Schema::new(cols.clone()); + let accelerable = conjuncts + .iter() + .any(|c| refs_in_schema(c, &schema) && is_accelerable(&def, c).unwrap_or(false)); + let est = catalog::load_stats(db, &def.name) + .await? + .map(|stats| estimate_filtered_rows(&stats, conjuncts)) + .unwrap_or(u64::MAX); + candidates.push(Candidate { + relation: rel, + def, + cols, + est, + accelerable, + }); } - // Start from the smallest relation. - let mut remaining: Vec = (0..loaded.len()).collect(); - remaining.sort_by_key(|&i| loaded[i].est); + // Prefer a relation with an indexable local predicate even before ANALYZE has + // produced statistics. Loading it is itself an index lookup and gives the + // exact (usually tiny) driving cardinality. + let mut remaining: Vec = (0..candidates.len()).collect(); + remaining.sort_by_key(|&i| (!candidates[i].accelerable, candidates[i].est)); let start = remaining.remove(0); - let mut cur_cols = std::mem::take(&mut loaded[start].cols); - let mut cur_rows = std::mem::take(&mut loaded[start].rows); + let (mut cur_cols, mut cur_rows) = + load_relation(db, vindex, candidates[start].relation, conjuncts).await?; + cur_rows = apply_pushdown(cur_rows, &cur_cols, conjuncts)?; while !remaining.is_empty() { // Among the remaining tables, pick the smallest one connected to what @@ -9788,15 +10402,15 @@ async fn build_inner_join_reordered( let mut best: Option<(usize, Expr)> = None; // (pos in remaining, connecting pred) let mut best_est = u64::MAX; for (pos, &i) in remaining.iter().enumerate() { - let t_aliases = relation_aliases(&loaded[i].cols); + let t_aliases = relation_aliases(&candidates[i].cols); for pred in &on_preds { if let Some((lq, rq)) = equi_qualifiers(pred) { let connects = (relation_aliases_contain(&cur_aliases, &lq) && relation_aliases_contain(&t_aliases, &rq)) || (relation_aliases_contain(&cur_aliases, &rq) && relation_aliases_contain(&t_aliases, &lq)); - if connects && loaded[i].est < best_est { - best_est = loaded[i].est; + if connects && candidates[i].est < best_est { + best_est = candidates[i].est; best = Some((pos, pred.clone())); break; } @@ -9807,9 +10421,46 @@ async fn build_inner_join_reordered( return Ok(None); }; let idx = remaining.remove(pos); - let rcols = std::mem::take(&mut loaded[idx].cols); - let rrows = std::mem::take(&mut loaded[idx].rows); + let candidate = &candidates[idx]; let left_schema = Schema::new(cur_cols); + let right_schema = Schema::new(candidate.cols.clone()); + + // A selective driver plus an indexed equality on the next table is the + // key case: fetch only matching partner rows instead of scanning and + // hashing the complete partner. Partner-local WHERE predicates remain + // pushdowns, and the full WHERE is still evaluated after the join. + if cur_rows.len() <= NLJ_MAX_DRIVING { + if let Some((driving_key, partner_col)) = equi_nlj(&pred, &left_schema, &right_schema) { + if candidate.def.pk_cols == [partner_col] + || index::index_on(&candidate.def, partner_col).is_some() + { + let mut out = Vec::new(); + let mut check = db.cancel_check(); + for left in &cur_rows { + check.tick()?; + let key = predicate::eval_row(&driving_key, &left_schema, left)?; + if key.is_null() { + continue; + } + let matches = + lookup_rows_by_eq(db, &candidate.def, partner_col, &key).await?; + for partner in apply_pushdown(matches, &candidate.cols, conjuncts)? { + let mut combined = Vec::with_capacity(left.len() + partner.len()); + combined.extend_from_slice(left); + combined.extend(partner); + out.push(combined); + } + } + cur_cols = left_schema.columns; + cur_cols.extend(candidate.cols.clone()); + cur_rows = out; + continue; + } + } + } + + let (rcols, mut rrows) = load_relation(db, vindex, candidate.relation, conjuncts).await?; + rrows = apply_pushdown(rrows, &rcols, conjuncts)?; let right_schema = Schema::new(rcols); let cancel = db.cancel_token(); let (c, r) = cpu_bound(|| { @@ -11499,12 +12150,500 @@ fn in_subquery_max() -> usize { env_usize("ELYRASQL_IN_SUBQUERY_MAX", 1_000_000) } -/// Max distinct rows `SELECT DISTINCT` may buffer (`ELYRASQL_DISTINCT_MAX`, -/// default 5,000,000) before erroring fail-safe rather than risking OOM. +/// Max distinct rows `SELECT DISTINCT` keeps in its in-memory fast path +/// (`ELYRASQL_DISTINCT_MAX`, default 5,000,000) before spilling to disk. fn distinct_max() -> usize { env_usize("ELYRASQL_DISTINCT_MAX", 5_000_000) } +/// Approximate byte budget for the in-memory DISTINCT fast path. Row-count +/// limits alone are unsafe for wide projections. +fn distinct_max_bytes() -> usize { + env_usize("ELYRASQL_DISTINCT_MAX_BYTES", 256 << 20) +} + +fn distinct_resident_exceeds( + rows: usize, + bytes: usize, + row_limit: usize, + byte_limit: usize, +) -> bool { + rows > row_limit.max(1) || bytes > byte_limit +} + +async fn run_distinct_blocking(context: &'static str, work: F) -> Result +where + F: FnOnce() -> Result + Send + 'static, + T: Send + 'static, +{ + tokio::task::spawn_blocking(work) + .await + .map_err(|error| Error::Storage(format!("DISTINCT {context} worker failed: {error}")))? +} + +fn distinct_row_key( + row: &[Value], + schema: &Schema, + collations: &[elyra_core::Collation], +) -> Vec { + let mut key = Vec::with_capacity(row.len() * 9); + for (index, value) in row.iter().enumerate() { + let collation = collations.get(index).copied().unwrap_or_default(); + if matches!( + schema.columns.get(index).map(|column| &column.ty), + Some(ColumnType::Float) + ) { + if let Some(number) = value.as_mysql_f64() { + Value::Float(number).push_collation_key_coll(&mut key, collation); + continue; + } + } + value.push_collation_key_coll(&mut key, collation); + } + key +} + +/// Deduplicate a projected stream while preserving the first representative of +/// each collation-aware value and the inner query's row order. +/// +/// Small results stay on the hash-set fast path. Once `memory_rows` distinct +/// values have accumulated, the retained representatives and all later input +/// are externally sorted by value and ordinal using the independent ORDER BY +/// memory budget. The first ordinal in each value group is then sorted back +/// into input order. Bounded results stay literal; unbounded results are written +/// directly to the frame format consumed by [`RowStream::spill`]. +async fn distinct_rows( + mut input: RowStream, + offset: usize, + limit: Option, + memory_rows: usize, + cancel: std::sync::Arc, +) -> Result { + use std::io::{BufWriter, Write}; + + let schema = input.schema.clone(); + let collations: Vec = schema + .columns + .iter() + .map(|column| column.collation) + .collect(); + let mut seen = std::collections::HashSet::new(); + let mut resident: Vec<(u64, Vec)> = Vec::new(); + let mut resident_bytes = 0usize; + let mut value_sorter = None; + let mut ordinal = 0u64; + let mut check = elyra_core::cancel::CancelCheck::new(cancel.clone()); + check.tick_now()?; + + loop { + let batch = input.next_batch(8192).await?; + if batch.is_empty() { + break; + } + if let Some(mut sorter) = value_sorter.take() { + let batch_len = u64::try_from(batch.len()) + .map_err(|_| Error::Query("SELECT DISTINCT batch is too large".into()))?; + let start_ordinal = ordinal; + ordinal = ordinal.checked_add(batch_len).ok_or_else(|| { + Error::Query("SELECT DISTINCT input row ordinal overflowed".into()) + })?; + let collations = collations.clone(); + let distinct_schema = schema.clone(); + let blocking_cancel = cancel.clone(); + sorter = run_distinct_blocking("spill", move || -> Result { + let mut check = elyra_core::cancel::CancelCheck::new(blocking_cancel); + for (position, row) in batch.into_iter().enumerate() { + check.tick()?; + push_distinct_candidate( + &mut sorter, + row, + start_ordinal + position as u64, + &distinct_schema, + &collations, + )?; + } + Ok(sorter) + }) + .await?; + value_sorter = Some(sorter); + continue; + } + for row in batch { + check.tick()?; + if let Some(sorter) = &mut value_sorter { + push_distinct_candidate(sorter, row, ordinal, &schema, &collations)?; + } else { + let key = distinct_row_key(&row, &schema, &collations); + let key_len = key.len(); + if seen.insert(key) { + resident_bytes = resident_bytes + .saturating_add(key_len.saturating_add(estimated_row_bytes(&row))); + resident.push((ordinal, row)); + if distinct_resident_exceeds( + resident.len(), + resident_bytes, + memory_rows, + distinct_max_bytes(), + ) { + let mut sorter = crate::sort::Sorter::new( + vec![true, true], + vec![elyra_core::Collation::Bin; 2], + 0, + None, + crate::sort::sort_max_rows(), + ); + for (resident_ordinal, resident_row) in resident.drain(..) { + push_distinct_candidate( + &mut sorter, + resident_row, + resident_ordinal, + &schema, + &collations, + )?; + } + seen.clear(); + seen.shrink_to_fit(); + resident_bytes = 0; + value_sorter = Some(sorter); + } + } + } + ordinal = ordinal.checked_add(1).ok_or_else(|| { + Error::Query("SELECT DISTINCT input row ordinal overflowed".into()) + })?; + } + } + + let Some(mut value_sorter) = value_sorter else { + let mut rows = resident.into_iter().map(|(_, row)| row).collect(); + apply_offset_limit(&mut rows, offset, limit); + return Ok(RowStream::literal(schema, rows)); + }; + + let blocking_cancel = cancel.clone(); + let distinct_schema = schema.clone(); + let order_sorter = run_distinct_blocking("merge", move || -> Result { + let mut check = elyra_core::cancel::CancelCheck::new(blocking_cancel); + let mut order_sorter = crate::sort::Sorter::new( + vec![true], + vec![elyra_core::Collation::Bin], + offset, + limit, + crate::sort::sort_max_rows(), + ); + let mut previous_key = None; + value_sorter.finish_with(|mut candidate| { + check.tick()?; + let candidate_ordinal = candidate + .pop() + .ok_or_else(|| Error::Storage("DISTINCT spill row missing ordinal".into()))?; + let key = distinct_row_key(&candidate, &distinct_schema, &collations); + if previous_key.as_ref() != Some(&key) { + previous_key = Some(key); + order_sorter.push(vec![candidate_ordinal], candidate)?; + } + Ok(()) + })?; + Ok(order_sorter) + }) + .await?; + + if limit.is_some_and(|limit| limit <= crate::sort::sort_max_rows()) { + let rows = run_distinct_blocking("order", move || { + let mut order_sorter = order_sorter; + order_sorter.finish() + }) + .await?; + return Ok(RowStream::literal(schema, rows)); + } + + let columns = schema.columns.len(); + let (path, file, numeric_types) = run_distinct_blocking("output", move || { + let (path, file) = create_distinct_spill()?; + let mut writer = BufWriter::new(file); + let mut numeric_types = crate::stream::NumericTypeReconciler::new(columns); + let mut order_sorter = order_sorter; + let mut check = elyra_core::cancel::CancelCheck::new(cancel); + let write_result = order_sorter.finish_with(|row| { + check.tick()?; + numeric_types.observe(&row); + let frame = + bincode::serialize(&row).map_err(|error| Error::Storage(error.to_string()))?; + if frame.len() > elyra_core::max_frame_bytes() || frame.len() > u32::MAX as usize { + return Err(Error::Storage("DISTINCT spill row frame too large".into())); + } + writer.write_all(&(frame.len() as u32).to_le_bytes())?; + writer.write_all(&frame)?; + Ok(()) + }); + if let Err(error) = write_result { + drop(writer); + let _ = std::fs::remove_file(&path); + return Err(error); + } + if let Err(error) = writer.flush() { + drop(writer); + let _ = std::fs::remove_file(&path); + return Err(Error::Io(error)); + } + let file = writer.into_inner().map_err(|error| { + let _ = std::fs::remove_file(&path); + Error::Io(error.into_error()) + })?; + Ok((path, file, numeric_types)) + }) + .await?; + let mut schema = schema; + numeric_types.reconcile(&mut schema); + RowStream::spill(schema, path, file) +} + +fn push_distinct_candidate( + sorter: &mut crate::sort::Sorter, + row: Vec, + ordinal: u64, + schema: &Schema, + collations: &[elyra_core::Collation], +) -> Result<()> { + let keys = vec![ + Value::Bytes(distinct_row_key(&row, schema, collations)), + Value::UInt(ordinal), + ]; + let mut candidate = row; + candidate.push(Value::UInt(ordinal)); + sorter.push(keys, candidate) +} + +fn create_distinct_spill() -> Result<(std::path::PathBuf, std::fs::File)> { + static SEQUENCE: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + + loop { + let sequence = SEQUENCE.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "elyrasql-sort-{}-distinct-{sequence}.tmp", + std::process::id() + )); + let mut options = std::fs::OpenOptions::new(); + options.read(true).write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + match options.open(&path) { + Ok(file) => return Ok((path, file)), + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => continue, + Err(error) => return Err(Error::Io(error)), + } + } +} + +#[cfg(test)] +mod distinct_spill_tests { + use super::*; + + #[test] + fn wide_rows_spill_before_the_row_count_limit() { + assert!(distinct_resident_exceeds(2, 101, 1_000, 100)); + assert!(!distinct_resident_exceeds(2, 100, 1_000, 100)); + } + + #[tokio::test] + async fn spill_work_runs_off_the_async_runtime_thread() { + let runtime_thread = std::thread::current().id(); + let worker_thread = run_distinct_blocking("test", || Ok(std::thread::current().id())) + .await + .unwrap(); + assert_ne!(worker_thread, runtime_thread); + } + + async fn collect(mut stream: RowStream) -> Vec> { + let mut rows = Vec::new(); + loop { + let batch = stream.next_batch(2).await.unwrap(); + if batch.is_empty() { + return rows; + } + rows.extend(batch); + } + } + + fn text_stream(values: &[&str]) -> RowStream { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Text, false)]); + let rows = values + .iter() + .map(|value| vec![Value::Text((*value).into())]) + .collect(); + RowStream::literal(schema, rows) + } + + #[tokio::test] + async fn spilled_distinct_preserves_first_representative_and_input_order() { + let stream = distinct_rows( + text_stream(&["z", "A", "z", "b", "a", "c", "B"]), + 0, + None, + 2, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(); + + assert_eq!( + collect(stream).await, + vec![ + vec![Value::Text("z".into())], + vec![Value::Text("A".into())], + vec![Value::Text("b".into())], + vec![Value::Text("c".into())], + ] + ); + } + + #[tokio::test] + async fn spilled_distinct_applies_offset_and_limit_after_deduplication() { + let stream = distinct_rows( + text_stream(&["z", "A", "z", "b", "a", "c", "B"]), + 1, + Some(2), + 2, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(); + + assert_eq!( + collect(stream).await, + vec![vec![Value::Text("A".into())], vec![Value::Text("b".into())]] + ); + } + + #[tokio::test] + async fn spilled_distinct_groups_by_the_canonical_key_not_sort_equality() { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Float, false)]); + let input = RowStream::literal( + schema, + vec![ + vec![Value::Int(1)], + vec![Value::Float(1.0)], + vec![Value::Int(1)], + ], + ); + let stream = distinct_rows( + input, + 0, + None, + 1, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(); + + assert_eq!(stream.schema.columns[0].ty, ColumnType::Int); + assert_eq!(collect(stream).await, vec![vec![Value::Int(1)]]); + } + + #[cfg(unix)] + #[test] + fn distinct_spill_is_owner_only() { + use std::os::unix::fs::PermissionsExt; + + let (path, file) = create_distinct_spill().unwrap(); + let mode = file.metadata().unwrap().permissions().mode() & 0o777; + drop(file); + let _ = std::fs::remove_file(path); + assert_eq!(mode, 0o600); + } + + #[tokio::test] + async fn thresholds_zero_and_one_match_in_memory_for_mixed_rows() { + let mut ci = ColumnDef::new("ci", ColumnType::Text, true); + ci.collation = elyra_core::Collation::Ci; + let mut bin = ColumnDef::new("bin", ColumnType::Text, true); + bin.collation = elyra_core::Collation::Bin; + let schema = Schema::new(vec![ci, bin, ColumnDef::new("n", ColumnType::Float, true)]); + let rows = vec![ + vec![ + Value::Text("A".into()), + Value::Text("x".into()), + Value::Int(1), + ], + vec![ + Value::Text("a".into()), + Value::Text("x".into()), + Value::Int(1), + ], + vec![ + Value::Text("a".into()), + Value::Text("X".into()), + Value::Int(1), + ], + vec![Value::Null, Value::Null, Value::Decimal(100, 2)], + vec![Value::Null, Value::Null, Value::Decimal(100, 2)], + vec![ + Value::Text("b".into()), + Value::Text("x".into()), + Value::Float(1.0), + ], + ]; + + let expected = collect( + distinct_rows( + RowStream::literal(schema.clone(), rows.clone()), + 1, + Some(3), + usize::MAX, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(), + ) + .await; + for threshold in [0, 1] { + let actual = collect( + distinct_rows( + RowStream::literal(schema.clone(), rows.clone()), + 1, + Some(3), + threshold, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(), + ) + .await; + assert_eq!(actual, expected, "threshold={threshold}"); + } + } + + #[tokio::test] + async fn empty_distinct_preserves_schema_and_limit_zero_is_empty() { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Float, true)]); + let stream = distinct_rows( + RowStream::literal(schema, Vec::new()), + usize::MAX, + Some(0), + 0, + std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()), + ) + .await + .unwrap(); + assert_eq!(stream.schema.columns[0].ty, ColumnType::Float); + assert!(collect(stream).await.is_empty()); + } + + #[tokio::test] + async fn distinct_observes_preexisting_cancellation() { + let cancel = std::sync::Arc::new(elyra_core::cancel::QueryCancel::new()); + cancel.cancel(); + let error = match distinct_rows(text_stream(&["a", "b"]), 0, None, 1, cancel).await { + Ok(_) => panic!("cancelled DISTINCT unexpectedly succeeded"), + Err(error) => error, + }; + assert!(error.to_string().contains("query cancelled")); + } +} + /// Max rows a **materialising** join may buffer (`ELYRASQL_JOIN_MAX_ROWS`, default /// 10,000,000) before erroring fail-safe rather than risking OOM. /// @@ -11786,7 +12925,7 @@ fn accelerable(def: &TableDef, filter: Option<&Expr>) -> Result { if in_list_lookup(def, filter)?.is_some() { return Ok(true); } - Ok(range_bounds(def, filter)?.is_some()) + Ok(composite_range_bounds(def, filter)?.is_some() || range_bounds(def, filter)?.is_some()) } /// Collect `(storage_key, row)` for every row matching `filter`, up to @@ -12018,8 +13157,28 @@ async fn collect_matches_inner( } } - // Range fast path: `col > x` / `BETWEEN` on a PK or indexed column uses an - // ordered range scan, then re-applies the full filter. + // Composite secondary-index range: equality on a non-empty leading prefix, + // then a range on the immediately following column. + if let Some(query) = composite_range_bounds(def, filter)? { + let budget = index_range_budget(db, def).await?; + if let Some(candidates) = composite_index_range(db, def, &query, budget).await? { + for (key, row) in candidates { + if recheck(&row)? { + out.push((key, row)); + if limit.is_some_and(|limit| out.len() >= limit) { + return Ok(Some(out)); + } + } + } + return Ok(Some(out)); + } + if bail_on_wide_range { + return Ok(None); + } + } + + // Range fast path: `col > x` / `BETWEEN` on a PK or single-column index + // uses an ordered range scan, then re-applies the full filter. if let Some(rq) = range_bounds(def, filter)? { // A clustered (primary-key) range is a sequential read, so it is always // worth taking. A *secondary* index range pays a random fetch per row, so it @@ -14615,18 +15774,18 @@ fn compute_partition( let count_star = name == "count" && args.is_empty(); let arg0 = args.first().copied(); let n = idxs.len(); + let aggregate = WindowAggregate::new(name, count_star, arg0, rows, schema, idxs)?; match frame_mode(frame, ordered)? { FrameMode::Rows => { let f = frame.expect("rows frame present"); for (p, &i) in idxs.iter().enumerate() { let (lo, hi) = rows_bounds(f, p, n, schema, rows, idxs)?; - let members: &[usize] = if lo <= hi { &idxs[lo..=hi] } else { &[] }; - result[i] = window_agg(name, count_star, members, arg0, rows, schema)?; + result[i] = aggregate.evaluate(lo, hi, idxs, rows, schema)?; } } FrameMode::Whole => { - let agg = window_agg(name, count_star, idxs, arg0, rows, schema)?; + let agg = aggregate.evaluate(0, n.saturating_sub(1), idxs, rows, schema)?; for &i in idxs { result[i] = agg.clone(); } @@ -14639,13 +15798,69 @@ fn compute_partition( while q < n && order_key(idxs[q])? == key { q += 1; } - let agg = window_agg(name, count_star, &idxs[0..q], arg0, rows, schema)?; + let agg = aggregate.evaluate(0, q - 1, idxs, rows, schema)?; for &i in &idxs[p..q] { result[i] = agg.clone(); } p = q; } } + FrameMode::Range => { + let f = frame.expect("range frame present"); + if order.len() != 1 { + return Err(Error::Unsupported( + "RANGE offset frames require exactly one numeric ORDER BY expression" + .into(), + )); + } + let keys: Vec = idxs + .iter() + .map(|&i| predicate::eval_row(&order[0].0, schema, &rows[i])) + .collect::>()?; + let numeric_keys = keys + .iter() + .map(|key| { + if key.is_null() { + Ok(None) + } else { + RangeNumeric::from_value(key).map(Some) + } + }) + .collect::>>()?; + let (peer_lows, peer_highs) = peer_bounds(&keys); + let start_offset = frame_offset_value(&f.start_bound, schema, rows, idxs)?; + let end_offset = f + .end_bound + .as_ref() + .map(|bound| frame_offset_value(bound, schema, rows, idxs)) + .transpose()? + .flatten(); + let bounds = WindowRangeBounds { + frame: f, + keys: &keys, + numeric_keys: &numeric_keys, + peer_lows: &peer_lows, + peer_highs: &peer_highs, + ascending: order[0].1, + start_offset: start_offset.as_ref(), + end_offset: end_offset.as_ref(), + }; + for (p, &i) in idxs.iter().enumerate() { + let (lo, hi) = window_range_bounds(&bounds, p)?; + result[i] = aggregate.evaluate(lo, hi, idxs, rows, schema)?; + } + } + FrameMode::Groups => { + let f = frame.expect("groups frame present"); + let keys: Vec> = + idxs.iter().map(|&i| order_key(i)).collect::>()?; + let (group_starts, row_groups) = peer_groups(&keys); + for (p, &i) in idxs.iter().enumerate() { + let (lo, hi) = + groups_bounds(f, row_groups[p], &group_starts, n, schema, rows, idxs)?; + result[i] = aggregate.evaluate(lo, hi, idxs, rows, schema)?; + } + } } } "ntile" => { @@ -14729,6 +15944,8 @@ enum FrameMode { Rows, Whole, PeerRunning, + Range, + Groups, } /// Decide how to evaluate a framed aggregate. Explicit `ROWS` frames use @@ -14754,16 +15971,385 @@ fn frame_mode(frame: Option<&sqlparser::ast::WindowFrame>, ordered: bool) -> Res Ok(FrameMode::Whole) } else if running && ordered { Ok(FrameMode::PeerRunning) - } else if !ordered { + } else if running { Ok(FrameMode::Whole) + } else if matches!(f.units, U::Range) { + if ordered { + Ok(FrameMode::Range) + } else { + Err(Error::Unsupported( + "RANGE offset frames require exactly one numeric ORDER BY expression" + .into(), + )) + } } else { - Err(Error::Unsupported( - "only RANGE UNBOUNDED PRECEDING .. CURRENT ROW / UNBOUNDED FOLLOWING frames are supported" - .into(), - )) + Ok(FrameMode::Groups) + } + } + } +} + +fn frame_offset_value( + bound: &sqlparser::ast::WindowFrameBound, + schema: &Schema, + rows: &[Vec], + idxs: &[usize], +) -> Result> { + use sqlparser::ast::WindowFrameBound as B; + use sqlparser::ast::{Visit, Visitor}; + use std::ops::ControlFlow; + + struct NonConstantFinder; + impl Visitor for NonConstantFinder { + type Break = (); + + fn pre_visit_expr(&mut self, expr: &Expr) -> ControlFlow { + if matches!( + expr, + Expr::Identifier(_) + | Expr::CompoundIdentifier(_) + | Expr::Function(_) + | Expr::Subquery(_) + | Expr::Exists { .. } + ) { + ControlFlow::Break(()) + } else { + ControlFlow::Continue(()) + } + } + } + + let expr = match bound { + B::Preceding(Some(expr)) | B::Following(Some(expr)) => expr, + _ => return Ok(None), + }; + if expr.visit(&mut NonConstantFinder).is_break() { + return Err(Error::Query( + "window frame offsets must be constant expressions".into(), + )); + } + let value = predicate::eval_row(expr, schema, &rows[idxs[0]])?; + Ok(Some(value)) +} + +#[derive(Clone, Copy)] +struct FixedNumeric { + raw: i128, + scale: u8, +} + +#[derive(Clone, Copy)] +enum RangeNumeric { + Fixed(FixedNumeric), + Float(f64), +} + +impl RangeNumeric { + fn from_value(value: &Value) -> Result { + match value { + Value::Int(value) => Ok(Self::Fixed(FixedNumeric { + raw: i128::from(*value), + scale: 0, + })), + Value::UInt(value) => Ok(Self::Fixed(FixedNumeric { + raw: i128::from(*value), + scale: 0, + })), + Value::Decimal(raw, scale) => Ok(Self::Fixed(FixedNumeric { + raw: *raw, + scale: *scale, + })), + Value::Float(value) if value.is_finite() => Ok(Self::Float(*value)), + _ => Err(Error::Unsupported( + "RANGE offset frames require numeric ORDER BY values and offsets; temporal offsets are not supported" + .into(), + )), + } + } + + fn non_negative(self) -> bool { + match self { + Self::Fixed(value) => value.raw >= 0, + Self::Float(value) => value >= 0.0, + } + } + + fn shifted(self, offset: Self, add: bool) -> Result { + match (self, offset) { + (Self::Fixed(left), Self::Fixed(right)) => { + let scale = left.scale.max(right.scale); + let left = scale_fixed(left, scale)?; + let right = scale_fixed(right, scale)?; + let raw = if add { + left.checked_add(right) + } else { + left.checked_sub(right) + } + .ok_or_else(|| Error::Unsupported("RANGE numeric boundary overflow".into()))?; + Ok(Self::Fixed(FixedNumeric { raw, scale })) + } + (Self::Float(left), Self::Float(right)) => Ok(Self::Float(if add { + left + right + } else { + left - right + })), + (Self::Float(left), Self::Fixed(right)) => { + let divisor = 10f64.powi(i32::from(right.scale)); + Ok(Self::Float(if add { + left + right.raw as f64 / divisor + } else { + left - right.raw as f64 / divisor + })) + } + (Self::Fixed(_), Self::Float(_)) => Err(Error::Unsupported( + "floating RANGE offsets are not supported for exact integer or DECIMAL ordering keys" + .into(), + )), + } + } + + fn compare(self, other: Self) -> Result { + match (self, other) { + (Self::Fixed(left), Self::Fixed(right)) => { + let scale = left.scale.max(right.scale); + Ok(scale_fixed(left, scale)?.cmp(&scale_fixed(right, scale)?)) + } + (Self::Float(left), Self::Float(right)) => Ok(left.total_cmp(&right)), + (Self::Float(left), Self::Fixed(right)) => { + Ok(left.total_cmp(&(right.raw as f64 / 10f64.powi(i32::from(right.scale))))) + } + (Self::Fixed(_), Self::Float(_)) => Err(Error::Unsupported( + "mixed floating and exact RANGE ordering values are not supported".into(), + )), + } + } +} + +fn scale_fixed(value: FixedNumeric, scale: u8) -> Result { + let factor = 10_i128 + .checked_pow(u32::from(scale - value.scale)) + .ok_or_else(|| Error::Unsupported("RANGE decimal scale overflow".into()))?; + value + .raw + .checked_mul(factor) + .ok_or_else(|| Error::Unsupported("RANGE decimal value overflow".into())) +} + +#[derive(Clone, Copy)] +struct WindowRangeBounds<'a> { + frame: &'a sqlparser::ast::WindowFrame, + keys: &'a [Value], + numeric_keys: &'a [Option], + peer_lows: &'a [usize], + peer_highs: &'a [usize], + ascending: bool, + start_offset: Option<&'a Value>, + end_offset: Option<&'a Value>, +} + +fn window_range_bounds(bounds: &WindowRangeBounds<'_>, p: usize) -> Result<(usize, usize)> { + use sqlparser::ast::WindowFrameBound as B; + let WindowRangeBounds { + frame, + keys, + numeric_keys, + peer_lows, + peer_highs, + ascending, + start_offset, + end_offset, + } = *bounds; + for offset in [start_offset, end_offset].into_iter().flatten() { + if !RangeNumeric::from_value(offset)?.non_negative() { + return Err(Error::Query( + "window frame offsets must be non-negative numeric constants".into(), + )); + } + } + let current = &keys[p]; + if current.is_null() { + let (peer_lo, peer_hi) = (peer_lows[p], peer_highs[p]); + let null_boundary = |bound: &B, start: bool| match bound { + B::Preceding(None) => 0, + B::Following(None) => keys.len().saturating_sub(1), + B::CurrentRow | B::Preceding(Some(_)) | B::Following(Some(_)) => { + if start { + peer_lo + } else { + peer_hi + } + } + }; + return Ok(( + null_boundary(&frame.start_bound, true), + null_boundary(frame.end_bound.as_ref().unwrap_or(&B::CurrentRow), false), + )); + } + let current = RangeNumeric::from_value(current)?; + let boundary = |bound: &B, offset: Option<&Value>, start: bool| -> Result { + match bound { + B::Preceding(None) => Ok(0), + B::Following(None) => Ok(keys.len().saturating_sub(1)), + B::CurrentRow => Ok(if start { peer_lows[p] } else { peer_highs[p] }), + B::Preceding(Some(_)) | B::Following(Some(_)) => { + let offset = RangeNumeric::from_value( + offset.ok_or_else(|| Error::Query("window frame offset is missing".into()))?, + )?; + let add = matches!(bound, B::Following(_)) == ascending; + let target = current.shifted(offset, add)?; + if start { + range_lower_bound(numeric_keys, target, ascending) + } else { + range_upper_bound(numeric_keys, target, ascending) + } + } + } + }; + let lo = boundary(&frame.start_bound, start_offset, true)?; + let hi = boundary( + frame.end_bound.as_ref().unwrap_or(&B::CurrentRow), + end_offset, + false, + )?; + if hi == usize::MAX { + return Ok((1, 0)); + } + Ok((lo, hi)) +} + +fn peer_bounds(keys: &[T]) -> (Vec, Vec) { + let mut lows = vec![0; keys.len()]; + let mut highs = vec![0; keys.len()]; + let mut start = 0; + while start < keys.len() { + let mut end = start + 1; + while end < keys.len() && keys[end] == keys[start] { + end += 1; + } + for position in start..end { + lows[position] = start; + highs[position] = end - 1; + } + start = end; + } + (lows, highs) +} + +fn range_lower_bound( + keys: &[Option], + target: RangeNumeric, + ascending: bool, +) -> Result { + let mut lo = 0; + let mut hi = keys.len(); + while lo < hi { + let mid = lo + (hi - lo) / 2; + let before = match keys[mid] { + None => ascending, + Some(value) => { + let ordering = value.compare(target)?; + if ascending { + ordering.is_lt() + } else { + ordering.is_gt() + } + } + }; + if before { + lo = mid + 1; + } else { + hi = mid; + } + } + Ok(lo) +} + +fn range_upper_bound( + keys: &[Option], + target: RangeNumeric, + ascending: bool, +) -> Result { + let mut lo = 0; + let mut hi = keys.len(); + while lo < hi { + let mid = lo + (hi - lo) / 2; + let after = match keys[mid] { + None => !ascending, + Some(value) => { + let ordering = value.compare(target)?; + if ascending { + ordering.is_gt() + } else { + ordering.is_lt() + } } + }; + if after { + hi = mid; + } else { + lo = mid + 1; + } + } + Ok(lo.checked_sub(1).unwrap_or(usize::MAX)) +} + +fn peer_groups(keys: &[T]) -> (Vec, Vec) { + let mut starts = Vec::new(); + let mut row_groups = Vec::with_capacity(keys.len()); + for (p, key) in keys.iter().enumerate() { + if p == 0 || key != &keys[p - 1] { + starts.push(p); } + row_groups.push(starts.len() - 1); + } + (starts, row_groups) +} + +#[allow(clippy::too_many_arguments)] +fn groups_bounds( + frame: &sqlparser::ast::WindowFrame, + group: usize, + starts: &[usize], + row_count: usize, + schema: &Schema, + rows: &[Vec], + idxs: &[usize], +) -> Result<(usize, usize)> { + use sqlparser::ast::WindowFrameBound as B; + let group_count = starts.len(); + let boundary_group = |bound: &B| -> Result { + let offset = match frame_offset_value(bound, schema, rows, idxs)? { + None => Some(0), + Some(Value::Int(value)) => isize::try_from(value).ok().filter(|v| *v >= 0), + Some(Value::UInt(value)) => isize::try_from(value).ok(), + Some(Value::Decimal(raw, scale)) => { + let divisor = 10_i128.checked_pow(u32::from(scale)); + divisor + .filter(|divisor| raw >= 0 && raw % divisor == 0) + .and_then(|divisor| isize::try_from(raw / divisor).ok()) + } + Some(_) => None, + } + .ok_or_else(|| { + Error::Query("GROUPS frame offsets must be exact non-negative integers".into()) + })?; + Ok(match bound { + B::Preceding(None) => 0, + B::Following(None) => group_count as isize - 1, + B::CurrentRow => group as isize, + B::Preceding(Some(_)) => (group as isize).checked_sub(offset).unwrap_or(isize::MIN), + B::Following(Some(_)) => (group as isize).checked_add(offset).unwrap_or(isize::MAX), + }) + }; + let lo_group = boundary_group(&frame.start_bound)?.max(0) as usize; + let hi_group = boundary_group(frame.end_bound.as_ref().unwrap_or(&B::CurrentRow))? + .min(group_count as isize - 1); + if hi_group < 0 || lo_group as isize > hi_group || lo_group >= group_count { + return Ok((1, 0)); } + let hi_group = hi_group as usize; + let hi = starts.get(hi_group + 1).copied().unwrap_or(row_count) - 1; + Ok((starts[lo_group], hi)) } /// Physical `[lo, hi]` bounds (inclusive, clamped) for a `ROWS` frame at sorted @@ -14803,6 +16389,260 @@ fn const_isize(e: &Expr, schema: &Schema, rows: &[Vec], idxs: &[usize]) - Ok(v.as_mysql_f64().unwrap_or(0.0) as isize) } +struct WindowAggregate<'a> { + name: &'a str, + count_star: bool, + arg: Option<&'a Expr>, + values: Vec, + non_null_count: Vec, + numeric_count: Vec, + non_integer_count: Vec, + integer_sums: Vec, + numeric_sums: NumericRangeSums, +} + +struct NumericRangeSums { + leaf_count: usize, + tree: Vec, +} + +impl NumericRangeSums { + fn new(values: &[Value]) -> Self { + let leaf_count = values.len().next_power_of_two().max(1); + let mut tree = vec![0.0; leaf_count * 2]; + for (index, value) in values.iter().enumerate() { + tree[leaf_count + index] = value.as_mysql_f64().unwrap_or(0.0); + } + for index in (1..leaf_count).rev() { + tree[index] = tree[index * 2] + tree[index * 2 + 1]; + } + Self { leaf_count, tree } + } + + fn sum(&self, mut start: usize, mut end: usize) -> f64 { + start += self.leaf_count; + end += self.leaf_count; + let mut left_sum = 0.0; + let mut right_sum = 0.0; + while start < end { + if start % 2 == 1 { + left_sum += self.tree[start]; + start += 1; + } + if end % 2 == 1 { + end -= 1; + right_sum += self.tree[end]; + } + start /= 2; + end /= 2; + } + left_sum + right_sum + } +} + +fn window_aggregate_is_incremental(name: &str) -> bool { + matches!(name, "sum" | "count" | "avg") +} + +impl<'a> WindowAggregate<'a> { + fn new( + name: &'a str, + count_star: bool, + arg: Option<&'a Expr>, + rows: &[Vec], + schema: &Schema, + idxs: &[usize], + ) -> Result { + let values = match arg { + Some(expr) => idxs + .iter() + .map(|&index| predicate::eval_row(expr, schema, &rows[index])) + .collect::>>()?, + None => vec![Value::Null; idxs.len()], + }; + let mut non_null_count = Vec::with_capacity(values.len() + 1); + let mut numeric_count = Vec::with_capacity(values.len() + 1); + let mut non_integer_count = Vec::with_capacity(values.len() + 1); + let mut integer_sums: Vec = Vec::with_capacity(values.len() + 1); + non_null_count.push(0); + numeric_count.push(0); + non_integer_count.push(0); + integer_sums.push(0); + for value in &values { + non_null_count.push( + non_null_count.last().copied().unwrap_or_default() + usize::from(!value.is_null()), + ); + numeric_count.push( + numeric_count.last().copied().unwrap_or_default() + + usize::from(value.as_mysql_f64().is_some()), + ); + non_integer_count.push( + non_integer_count.last().copied().unwrap_or_default() + + usize::from(!matches!(value, Value::Int(_) | Value::Null)), + ); + let integer = match value { + Value::Int(value) => i128::from(*value), + _ => 0, + }; + integer_sums.push( + integer_sums + .last() + .copied() + .unwrap_or_default() + .checked_add(integer) + .ok_or_else(|| Error::Query("window integer sum overflowed".into()))?, + ); + } + let numeric_sums = NumericRangeSums::new(&values); + Ok(Self { + name, + count_star, + arg, + values, + non_null_count, + numeric_count, + non_integer_count, + integer_sums, + numeric_sums, + }) + } + + fn evaluate( + &self, + lo: usize, + hi: usize, + idxs: &[usize], + rows: &[Vec], + schema: &Schema, + ) -> Result { + if lo > hi || lo >= self.values.len() { + return Ok(if self.name == "count" { + Value::Int(0) + } else { + Value::Null + }); + } + let end = hi.min(self.values.len() - 1) + 1; + let len = end - lo; + if self.count_star { + return Ok(Value::Int(len as i64)); + } + if !window_aggregate_is_incremental(self.name) { + return window_agg(self.name, false, &idxs[lo..end], self.arg, rows, schema); + } + match self.name { + "count" => Ok(Value::Int( + (self.non_null_count[end] - self.non_null_count[lo]) as i64, + )), + "sum" | "avg" => { + let count = self.numeric_count[end] - self.numeric_count[lo]; + if count == 0 { + return Ok(Value::Null); + } + if self.non_integer_count[end] != self.non_integer_count[lo] { + let sum = self.numeric_sums.sum(lo, end); + return Ok(if self.name == "avg" { + Value::Float(sum / count as f64) + } else { + Value::Float(sum) + }); + } + let integer_sum = self.integer_sums[end] - self.integer_sums[lo]; + if self.name == "avg" { + Ok(Value::Float(integer_sum as f64 / count as f64)) + } else { + i64::try_from(integer_sum) + .map(Value::Int) + .map_err(|_| Error::Query("window integer sum overflowed".into())) + } + } + _ => unreachable!("incremental window aggregate helper and dispatcher diverged"), + } + } +} + +#[cfg(test)] +mod window_incremental_tests { + use super::*; + + fn fixture() -> (Schema, Vec>, Expr, Vec) { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Int, true)]); + let rows = vec![ + vec![Value::Int(2)], + vec![Value::Null], + vec![Value::Int(5)], + vec![Value::Int(-1)], + ]; + let expression = Expr::Identifier(Ident::new("v")); + (schema, rows, expression, vec![0, 1, 2, 3]) + } + + #[test] + fn prefix_aggregates_preserve_null_and_empty_frame_semantics() { + let (schema, rows, expression, idxs) = fixture(); + let sum = + WindowAggregate::new("sum", false, Some(&expression), &rows, &schema, &idxs).unwrap(); + let count = + WindowAggregate::new("count", false, Some(&expression), &rows, &schema, &idxs).unwrap(); + let avg = + WindowAggregate::new("avg", false, Some(&expression), &rows, &schema, &idxs).unwrap(); + + assert_eq!( + sum.evaluate(1, 3, &idxs, &rows, &schema).unwrap(), + Value::Int(4) + ); + assert_eq!( + count.evaluate(1, 3, &idxs, &rows, &schema).unwrap(), + Value::Int(2) + ); + assert_eq!( + avg.evaluate(1, 3, &idxs, &rows, &schema).unwrap(), + Value::Float(2.0) + ); + assert_eq!( + sum.evaluate(2, 1, &idxs, &rows, &schema).unwrap(), + Value::Null + ); + assert_eq!( + count.evaluate(2, 1, &idxs, &rows, &schema).unwrap(), + Value::Int(0) + ); + } + + #[test] + fn integer_prefix_subtraction_is_exact_above_f64_precision() { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Int, false)]); + let rows = vec![vec![Value::Int(9_007_199_254_740_992)], vec![Value::Int(1)]]; + let expression = Expr::Identifier(Ident::new("v")); + let idxs = vec![0, 1]; + let sum = + WindowAggregate::new("sum", false, Some(&expression), &rows, &schema, &idxs).unwrap(); + + assert_eq!( + sum.evaluate(1, 1, &idxs, &rows, &schema).unwrap(), + Value::Int(1) + ); + } + + #[test] + fn floating_range_sum_avoids_prefix_cancellation() { + let schema = Schema::new(vec![ColumnDef::new("v", ColumnType::Float, false)]); + let rows = vec![ + vec![Value::Float(9_007_199_254_740_992.0)], + vec![Value::Float(1.0)], + ]; + let expression = Expr::Identifier(Ident::new("v")); + let idxs = vec![0, 1]; + let sum = + WindowAggregate::new("sum", false, Some(&expression), &rows, &schema, &idxs).unwrap(); + + assert_eq!( + sum.evaluate(1, 1, &idxs, &rows, &schema).unwrap(), + Value::Float(1.0) + ); + } +} + /// Aggregate `name` over the given member rows (evaluating `arg` per row). fn window_agg( name: &str, @@ -17308,10 +19148,21 @@ async fn correlated_select( let all = scan_rows(db, def, None).await?; let mut matched: Vec> = Vec::new(); + // A deliberately narrow semi/anti-join rewrite for the most common + // correlated shape. Anything that cannot be proven equivalent keeps the + // general per-row subquery path below. + let decorrelated = prepare_correlated_exists(db, corr_filter, def, outer).await?; + for row in all { - let bound = bind_outer(db, corr_filter, outer, &def.schema, &row); - let resolved = resolve_subqueries_with_outer(db, vindex, bound, &def.schema, &row).await?; - if predicate::matches(&resolved, &def.schema, &row)? { + let matches = if let Some(plan) = &decorrelated { + plan.matches(&def.schema, &row)? + } else { + let bound = bind_outer(db, corr_filter, outer, &def.schema, &row); + let resolved = + resolve_subqueries_with_outer(db, vindex, bound, &def.schema, &row).await?; + predicate::matches(&resolved, &def.schema, &row)? + }; + if matches { matched.push(row); } } @@ -17427,6 +19278,293 @@ async fn correlated_select( ))) } +struct CorrelatedExistsPlan { + outer_column: usize, + inner_keys: std::collections::HashSet>, + collation: elyra_core::Collation, + negated: bool, + residual: Option, +} + +impl CorrelatedExistsPlan { + fn matches(&self, schema: &Schema, outer_row: &[Value]) -> Result { + let member = key_bytes_coll(&outer_row[self.outer_column], self.collation) + .is_some_and(|key| self.inner_keys.contains(&key)); + if member == self.negated { + return Ok(false); + } + self.residual.as_ref().map_or(Ok(true), |residual| { + predicate::matches(residual, schema, outer_row) + }) + } +} + +/// Build a one-time semantic-key membership set for a safe correlated +/// `EXISTS`/`NOT EXISTS` semi-join. The accepted slice is intentionally strict: +/// one outer-table column equals one column of one plain inner table, with no +/// other inner clauses or query modifiers. Incompatible types/collations and +/// oversized inner inputs silently retain the nested-loop implementation. +async fn prepare_correlated_exists( + db: &Session, + filter: &Expr, + outer_def: &TableDef, + outer_qualifier: &[String], +) -> Result> { + let Some(shape) = + prepare_correlated_exists_shape(db, filter, outer_def, outer_qualifier).await? + else { + return Ok(None); + }; + let inner_rows = scan_rows(db, &shape.inner_def, None).await?; + if inner_rows.len() > in_subquery_max() { + return Ok(None); + } + let inner_keys = inner_rows + .iter() + .filter_map(|row| key_bytes_coll(&row[shape.inner_column], shape.collation)) + .collect(); + Ok(Some(CorrelatedExistsPlan { + outer_column: shape.outer_column, + inner_keys, + collation: shape.collation, + negated: shape.negated, + residual: shape.residual, + })) +} + +struct CorrelatedExistsShape { + outer_column: usize, + inner_column: usize, + inner_def: TableDef, + collation: elyra_core::Collation, + negated: bool, + residual: Option, +} + +/// Whether `filter` has the exact correlated semi/anti-membership shape used +/// by execution. This performs catalog/type analysis only; it does not scan or +/// execute the inner query, so callers such as `EXPLAIN` can use it safely. +pub(crate) async fn correlated_exists_membership_eligible( + db: &Session, + filter: &Expr, + outer_def: &TableDef, + outer_qualifier: &[String], +) -> Result { + Ok( + prepare_correlated_exists_shape(db, filter, outer_def, outer_qualifier) + .await? + .is_some(), + ) +} + +async fn prepare_correlated_exists_shape( + db: &Session, + filter: &Expr, + outer_def: &TableDef, + outer_qualifier: &[String], +) -> Result> { + let mut conjuncts = Vec::new(); + split_and(filter, &mut conjuncts); + let candidates: Vec<&Expr> = conjuncts + .iter() + .filter(|expr| { + matches!(expr, Expr::Exists { subquery, .. } if query_refs_qualifier(subquery, outer_qualifier)) + }) + .collect(); + let [exists] = candidates.as_slice() else { + return Ok(None); + }; + let Expr::Exists { subquery, negated } = *exists else { + return Ok(None); + }; + + if subquery.with.is_some() + || subquery.order_by.is_some() + || subquery.limit.is_some() + || !subquery.limit_by.is_empty() + || subquery.offset.is_some() + || subquery.fetch.is_some() + || !subquery.locks.is_empty() + || subquery.for_clause.is_some() + || subquery.settings.is_some() + || subquery.format_clause.is_some() + { + return Ok(None); + } + let SetExpr::Select(inner_select) = subquery.body.as_ref() else { + return Ok(None); + }; + let sqlparser::ast::GroupByExpr::Expressions(group_by, modifiers) = &inner_select.group_by + else { + return Ok(None); + }; + if inner_select.distinct.is_some() + || inner_select.top.is_some() + || inner_select.into.is_some() + || !inner_select.lateral_views.is_empty() + || inner_select.prewhere.is_some() + || !group_by.is_empty() + || !modifiers.is_empty() + || inner_select.having.is_some() + || !inner_select.cluster_by.is_empty() + || !inner_select.distribute_by.is_empty() + || !inner_select.sort_by.is_empty() + || !inner_select.named_window.is_empty() + || inner_select.qualify.is_some() + || inner_select.value_table_mode.is_some() + || inner_select.connect_by.is_some() + || inner_select.from.len() != 1 + || !inner_select.from[0].joins.is_empty() + || projection_correlated(&inner_select.projection, outer_qualifier) + || aggregate::projection_has_aggregate(&inner_select.projection) + || projection_has_window(&inner_select.projection) + { + return Ok(None); + } + let TableFactor::Table { + name: inner_name, + alias: inner_alias, + .. + } = &inner_select.from[0].relation + else { + return Ok(None); + }; + let Some(correlation) = inner_select.selection.as_ref() else { + return Ok(None); + }; + let Expr::BinaryOp { + left, + op: sqlparser::ast::BinaryOperator::Eq, + right, + } = correlation + else { + return Ok(None); + }; + + let inner_table = stored_table_ident(db, inner_name)?; + let inner_def = catalog::load(db, &inner_table).await?; + let inner_qualifier = factor_qualifier_object(db, &inner_select.from[0].relation) + .map(|qualifier| object_name_parts(&qualifier)) + .unwrap_or_else(|| { + inner_alias + .as_ref() + .map(|alias| vec![alias.name.value.clone()]) + .unwrap_or_else(|| vec![inner_table]) + }); + let pair = correlation_column_pair( + left, + right, + outer_qualifier, + &outer_def.schema, + &inner_qualifier, + &inner_def.schema, + ) + .or_else(|| { + correlation_column_pair( + right, + left, + outer_qualifier, + &outer_def.schema, + &inner_qualifier, + &inner_def.schema, + ) + }); + let Some((outer_column, inner_column)) = pair else { + return Ok(None); + }; + let outer_column_def = &outer_def.schema.columns[outer_column]; + let inner_column_def = &inner_def.schema.columns[inner_column]; + if outer_column_def.ty != inner_column_def.ty + || outer_column_def.collation != inner_column_def.collation + // SQL NaN equality is false while the grouping/hash key deliberately + // canonicalises NaNs. Vector equality likewise has no scalar-key + // contract. Keep both on the interpreter path. + || matches!(outer_column_def.ty, ColumnType::Float | ColumnType::Vector(_)) + { + return Ok(None); + } + + let collation = outer_column_def.collation; + + // The membership predicate is evaluated directly. Normalise the remaining + // outer-only conjuncts once so the hot row loop neither clones/maps the AST + // nor resolves a subquery. Any residual construct we cannot prove local to + // the outer schema keeps the general interpreter path. + let residual_conjuncts = conjuncts + .iter() + .filter(|conjunct| conjunct != exists) + .map(|conjunct| normalise_outer_references(conjunct, outer_qualifier)) + .collect::>(); + if residual_conjuncts + .iter() + .any(|conjunct| expr_has_subquery(conjunct) || !refs_in_schema(conjunct, &outer_def.schema)) + { + return Ok(None); + } + let residual = residual_conjuncts + .into_iter() + .reduce(|left, right| Expr::BinaryOp { + left: Box::new(left), + op: sqlparser::ast::BinaryOperator::And, + right: Box::new(right), + }); + Ok(Some(CorrelatedExistsShape { + outer_column, + inner_column, + inner_def, + collation, + negated: *negated, + residual, + })) +} + +fn normalise_outer_references(expr: &Expr, outer_qualifier: &[String]) -> Expr { + map_expr(expr, &|candidate| match candidate { + Expr::CompoundIdentifier(parts) + if parts.len() >= 2 + && qualifier_parts_match(outer_qualifier, &parts[..parts.len() - 1]) => + { + parts.last().cloned().map(Expr::Identifier) + } + _ => None, + }) +} + +fn correlation_column_pair( + outer_expr: &Expr, + inner_expr: &Expr, + outer_qualifier: &[String], + outer_schema: &Schema, + inner_qualifier: &[String], + inner_schema: &Schema, +) -> Option<(usize, usize)> { + let outer_column = qualified_column_index(outer_expr, outer_qualifier, outer_schema, false)?; + let inner_column = qualified_column_index(inner_expr, inner_qualifier, inner_schema, true)?; + Some((outer_column, inner_column)) +} + +fn qualified_column_index( + expr: &Expr, + qualifier: &[String], + schema: &Schema, + allow_bare: bool, +) -> Option { + let column = match expr { + Expr::Nested(inner) => return qualified_column_index(inner, qualifier, schema, allow_bare), + Expr::Identifier(identifier) if allow_bare => &identifier.value, + Expr::CompoundIdentifier(parts) + if parts.len() >= 2 && qualifier_parts_match(qualifier, &parts[..parts.len() - 1]) => + { + &parts.last()?.value + } + _ => return None, + }; + schema + .columns + .iter() + .position(|candidate| predicate::identifier_eq(&candidate.name, column)) +} + /// Rewrite qualified outer column references (`outer.col`) in `expr` to /// literals from `row`, including inside subqueries. Bare names remain bound /// to the innermost query scope. @@ -18577,10 +20715,12 @@ async fn olap_aggregate( if plan.is_count_star_only() { if let Some(n) = index_count_eq(db, def, f).await? { let mut agg = plan.new_aggregator(); - let empty: Vec = Vec::new(); - for _ in 0..n { - agg.feed(&empty); - } + agg.seed_count_star(n); + return Ok(agg); + } + if let Some(n) = index_count_composite_range(db, def, f).await? { + let mut agg = plan.new_aggregator(); + agg.seed_count_star(n); return Ok(agg); } } @@ -19854,6 +21994,67 @@ async fn index_count_eq(db: &Session, def: &TableDef, filter: &Expr) -> Result Result> { + let Some(query) = composite_range_bounds(def, Some(filter))? else { + return Ok(None); + }; + let range_column = query.index.cols[query.prefix.len()]; + let prefix_columns = &query.index.cols[..query.prefix.len()]; + let mut conjuncts = Vec::new(); + split_and(filter, &mut conjuncts); + let mut seen_prefix = std::collections::HashSet::new(); + for conjunct in &conjuncts { + if let Some((column, _)) = eq_col_literal(def, Some(conjunct))? { + if prefix_columns.contains(&column) && seen_prefix.insert(column) { + continue; + } + return Ok(None); + } + if as_range(def, conjunct)?.is_some_and(|(column, _, _)| column == range_column) + || as_between(def, conjunct)?.is_some_and(|(column, _, _)| column == range_column) + { + continue; + } + return Ok(None); + } + + let lo = query + .lo + .as_ref() + .map(|(value, inclusive)| (value, *inclusive)); + let hi = query + .hi + .as_ref() + .map(|(value, inclusive)| (value, *inclusive)); + if db.in_txn() { + return Ok(Some( + index::lookup_prefix_range(db, &def.name, query.index, &query.prefix, lo, hi) + .await? + .len() as u64, + )); + } + let Some((start, end)) = + index::prefix_range_scan_bounds(&def.name, query.index, &query.prefix, lo, hi)? + else { + return Ok(Some(0)); + }; + let count = db + .raw_db() + .scan_range_fold(start, end, 0u64, |count, _, _| { + *count += 1; + Ok(()) + }) + .await?; + Ok(Some(count)) +} + /// Collect the schema column indices referenced by `e` into `out`. Returns /// `false` if the expression contains any form we don't fully understand, in /// which case the caller must conservatively assume *all* columns are needed. diff --git a/crates/elyra-engine/src/index.rs b/crates/elyra-engine/src/index.rs index 0cca3a2..64b750f 100644 --- a/crates/elyra-engine/src/index.rs +++ b/crates/elyra-engine/src/index.rs @@ -7,7 +7,7 @@ //! ``` //! //! `enc(col_values)` is the order-preserving composite encoding of the indexed -//! columns, so equality and (single-column) range lookups are B-tree scans. +//! columns, so equality and left-prefix range lookups are B-tree scans. use crate::session::Session; use elyra_core::{Collation, Result, Value}; @@ -118,23 +118,25 @@ fn value_prefix( Ok(k) } -fn entry_key( +fn value_prefix_encoded(table: &str, index: &str, encoded: &[u8]) -> Vec { + let mut key = index_prefix(table, index); + key.extend_from_slice(encoded); + key.push(0); + key +} + +fn entry_key_encoded( table: &str, index: &str, - values: &[Value], - colls: &[Collation], - data_key: &[u8], + encoded: &[u8], + clustered_key: &[u8], unique: bool, -) -> Result> { - let mut k = value_prefix(table, index, values, colls)?; - // A UNIQUE index keys purely on the indexed values, so two rows with the - // same value collide (enforcing uniqueness). A non-unique index appends the - // clustered key so rows with equal values coexist. +) -> Vec { + let mut key = value_prefix_encoded(table, index, encoded); if !unique { - let clustered = &data_key[data_prefix(table).len()..]; - k.extend_from_slice(clustered); + key.extend_from_slice(clustered_key); } - Ok(k) + key } /// The probe key for a unique index's value tuple (== its entry key). A stored @@ -158,6 +160,7 @@ pub fn partition_entries_for_row( ) -> Result<(Vec, Vec)> { let mut nonuniq = Vec::new(); let mut uniq = Vec::new(); + let clustered_key = &data_key[b"data::".len() + def.name.len() + b"::".len()..]; for idx in &def.indexes { if idx.vector { continue; @@ -168,27 +171,23 @@ pub fn partition_entries_for_row( } continue; } - let values: Vec = idx.cols.iter().map(|&c| row[c].clone()).collect(); - if values.iter().any(|v| v.is_null()) || keyenc::encode_key(&values).is_err() { + let has_null = idx.cols.iter().any(|&column| row[column].is_null()); + let encoded = (!has_null) + .then(|| keyenc::encode_columns_coll(row, &idx.cols, &idx.col_collations).ok()) + .flatten(); + let Some(encoded) = encoded else { // Single-column NULL-indexing: record the NULL-keyed row under the // `indexnull::` keyspace (never unique -- NULLs don't collide). - if idx.indexes_nulls && idx.cols.len() == 1 && values[0].is_null() { + if idx.indexes_nulls && idx.cols.len() == 1 && row[idx.cols[0]].is_null() { nonuniq.push(( null_entry_key(&def.name, &idx.name, data_key), data_key.to_vec(), )); } continue; - } + }; let entry = ( - entry_key( - &def.name, - &idx.name, - &values, - &idx.col_collations, - data_key, - idx.unique, - )?, + entry_key_encoded(&def.name, &idx.name, &encoded, clustered_key, idx.unique), data_key.to_vec(), ); if idx.unique { @@ -208,16 +207,14 @@ pub fn unique_probe_keys(def: &TableDef, row: &[Value]) -> Result>> if idx.vector || !idx.unique { continue; } - let values: Vec = idx.cols.iter().map(|&c| row[c].clone()).collect(); - if values.iter().any(|v| v.is_null()) || keyenc::encode_key(&values).is_err() { + let has_null = idx.cols.iter().any(|&column| row[column].is_null()); + let encoded = (!has_null) + .then(|| keyenc::encode_columns_coll(row, &idx.cols, &idx.col_collations).ok()) + .flatten(); + let Some(encoded) = encoded else { continue; - } - out.push(unique_probe_key( - &def.name, - &idx.name, - &values, - &idx.col_collations, - )?); + }; + out.push(value_prefix_encoded(&def.name, &idx.name, &encoded)); } Ok(out) } @@ -235,6 +232,7 @@ pub fn entries_for_row( data_key: &[u8], ) -> Result, Vec)>> { let mut out = Vec::new(); + let clustered_key = &data_key[b"data::".len() + def.name.len() + b"::".len()..]; for idx in &def.indexes { if idx.vector { continue; // vector indexes are maintained separately @@ -245,26 +243,22 @@ pub fn entries_for_row( } continue; } - let values: Vec = idx.cols.iter().map(|&c| row[c].clone()).collect(); - if values.iter().any(|v| v.is_null()) || keyenc::encode_key(&values).is_err() { + let has_null = idx.cols.iter().any(|&column| row[column].is_null()); + let encoded = (!has_null) + .then(|| keyenc::encode_columns_coll(row, &idx.cols, &idx.col_collations).ok()) + .flatten(); + let Some(encoded) = encoded else { // Single-column NULL-indexing (see `partition_entries_for_row`). - if idx.indexes_nulls && idx.cols.len() == 1 && values[0].is_null() { + if idx.indexes_nulls && idx.cols.len() == 1 && row[idx.cols[0]].is_null() { out.push(( null_entry_key(&def.name, &idx.name, data_key), data_key.to_vec(), )); } continue; - } + }; out.push(( - entry_key( - &def.name, - &idx.name, - &values, - &idx.col_collations, - data_key, - idx.unique, - )?, + entry_key_encoded(&def.name, &idx.name, &encoded, clustered_key, idx.unique), data_key.to_vec(), )); } @@ -375,6 +369,89 @@ pub async fn lookup_range( Ok(keys) } +/// Range lookup on the column immediately following an equality-constrained +/// leading prefix of a composite index. Bounds are `(value, inclusive)`. +/// +/// `prefix_values` must correspond to the first N indexed columns and the +/// bounds to column N. Component encodings are self-delimiting, so the encoded +/// tuple prefix is also an exact byte prefix for every matching index entry. +pub async fn lookup_prefix_range( + db: &Session, + table: &str, + index: &IndexDef, + prefix_values: &[Value], + lo: Option<(&Value, bool)>, + hi: Option<(&Value, bool)>, +) -> Result>> { + let Some((mut start, end)) = prefix_range_scan_bounds(table, index, prefix_values, lo, hi)? + else { + return Ok(Vec::new()); + }; + + let mut keys = Vec::new(); + loop { + let batch = db + .scan_range(start.clone(), Some(end.clone()), 4096) + .await?; + if batch.is_empty() { + break; + } + let last = batch.len() < 4096; + start = batch + .last() + .map(|(key, _)| { + let mut next = key.clone(); + next.push(0); + next + }) + .expect("a non-empty range batch has a final key"); + keys.extend(batch.into_iter().map(|(_, data_key)| data_key)); + if last { + break; + } + } + Ok(keys) +} + +pub(crate) fn prefix_range_scan_bounds( + table: &str, + index: &IndexDef, + prefix_values: &[Value], + lo: Option<(&Value, bool)>, + hi: Option<(&Value, bool)>, +) -> Result, Vec)>> { + let mut equality_prefix = index_prefix(table, &index.name); + equality_prefix.extend_from_slice(&keyenc::encode_key_coll( + prefix_values, + &index.col_collations, + )?); + let range_collation = index + .col_collations + .get(prefix_values.len()) + .copied() + .unwrap_or_default(); + + let bound_prefix = |value: &Value| -> Result> { + let mut bound = equality_prefix.clone(); + bound.extend_from_slice(&keyenc::encode_coll(value, range_collation)?); + Ok(bound) + }; + let start = match lo { + Some((value, true)) => bound_prefix(value)?, + Some((value, false)) => prefix_upper_bound(&bound_prefix(value)?), + None => equality_prefix.clone(), + }; + let end = match hi { + Some((value, true)) => prefix_upper_bound(&bound_prefix(value)?), + Some((value, false)) => bound_prefix(value)?, + None => prefix_upper_bound(&equality_prefix), + }; + if start >= end { + return Ok(None); + } + Ok(Some((start, end))) +} + /// Smallest key strictly greater than every key with `prefix`. pub fn prefix_upper_bound(prefix: &[u8]) -> Vec { let mut end = prefix.to_vec(); diff --git a/crates/elyra-engine/src/keyenc.rs b/crates/elyra-engine/src/keyenc.rs index 9ac7560..e17454a 100644 --- a/crates/elyra-engine/src/keyenc.rs +++ b/crates/elyra-engine/src/keyenc.rs @@ -7,16 +7,6 @@ use elyra_core::{fold, Collation, Error, Result, Value}; -/// Encode a (possibly composite) key from its component values, in key order. -/// Text is case-folded (the default case-insensitive collation). -pub fn encode_key(values: &[Value]) -> Result> { - let mut out = Vec::with_capacity(values.len() * 8); - for v in values { - encode_component(v, Collation::Ci, &mut out)?; - } - Ok(out) -} - /// Encode a composite key honoring each component's collation (case-sensitive /// for `Bin`). `colls` is matched positionally; missing entries default to `Ci`. pub fn encode_key_coll(values: &[Value], colls: &[Collation]) -> Result> { @@ -28,6 +18,24 @@ pub fn encode_key_coll(values: &[Value], colls: &[Collation]) -> Result> Ok(out) } +/// Encode selected columns from a row without first cloning them into a +/// temporary value vector. `columns` and `colls` are both in key order. +pub fn encode_columns_coll( + row: &[Value], + columns: &[usize], + colls: &[Collation], +) -> Result> { + let mut out = Vec::with_capacity(columns.len() * 8); + for (position, &column) in columns.iter().enumerate() { + let value = row + .get(column) + .ok_or_else(|| Error::Storage("key column is outside the stored row".into()))?; + let collation = colls.get(position).copied().unwrap_or(Collation::Ci); + encode_component(value, collation, &mut out)?; + } + Ok(out) +} + /// Encode a single value (convenience for one-column keys/bounds). pub fn encode(value: &Value) -> Result> { let mut out = Vec::with_capacity(8); @@ -90,3 +98,23 @@ fn encode_component(value: &Value, coll: Collation, out: &mut Vec) -> Result pub fn encode_rowid(rowid: u64) -> [u8; 8] { rowid.to_be_bytes() } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn selected_column_encoding_matches_materialized_tuple_encoding() { + let row = vec![ + Value::Text("ignored".into()), + Value::Int(-7), + Value::Text("Case".into()), + ]; + let materialized = vec![row[2].clone(), row[1].clone()]; + let collations = [Collation::Bin, Collation::Ci]; + assert_eq!( + encode_columns_coll(&row, &[2, 1], &collations).unwrap(), + encode_key_coll(&materialized, &collations).unwrap() + ); + } +} diff --git a/crates/elyra-engine/src/lib.rs b/crates/elyra-engine/src/lib.rs index 2ef0fe6..3851e53 100644 --- a/crates/elyra-engine/src/lib.rs +++ b/crates/elyra-engine/src/lib.rs @@ -1075,7 +1075,11 @@ impl Engine { let content = tokio::fs::read_to_string(&spec.path).await.map_err(|e| { Error::Query(format!("LOAD DATA: cannot read '{}': {e}", spec.path)) })?; - let stmts = exec::build_load_inserts(&spec, &content, 1000); + // Keep LOAD DATA on the plain-INSERT writer fast path while + // amortising SQL parsing and durable commits over a genuinely + // bulk-sized unit. The builder still splits larger files so one + // statement cannot grow without bound. + let stmts = exec::build_load_inserts(&spec, &content, 50_000); let mut total = 0u64; for stmt in stmts { for r in Box::pin(self.execute_as(&stmt, privilege, user, sess)).await? { diff --git a/crates/elyra-engine/src/session.rs b/crates/elyra-engine/src/session.rs index 4b604bc..5303d3a 100644 --- a/crates/elyra-engine/src/session.rs +++ b/crates/elyra-engine/src/session.rs @@ -115,6 +115,33 @@ fn rollback_tx_to(tx: &mut TxnState, undo_mark: usize, ranges_len: usize) { tx.ranges.truncate(ranges_len); } +fn coalesce_ranges(mut ranges: Vec<(Vec, Option>)>) -> Vec<(Vec, Option>)> { + ranges.sort_unstable_by(|left, right| left.0.cmp(&right.0)); + let mut merged: Vec<(Vec, Option>)> = Vec::with_capacity(ranges.len()); + for (start, end) in ranges { + let Some((_, previous_end)) = merged.last_mut() else { + merged.push((start, end)); + continue; + }; + let overlaps = previous_end + .as_ref() + .is_none_or(|previous_end| start.as_slice() <= previous_end.as_slice()); + if !overlaps { + merged.push((start, end)); + continue; + } + match (&*previous_end, end) { + (None, _) => {} + (_, None) => *previous_end = None, + (Some(previous), Some(candidate)) if candidate > *previous => { + *previous_end = Some(candidate); + } + _ => {} + } + } + merged +} + pub struct Session { db: Db, txn: Mutex>, @@ -205,6 +232,16 @@ impl Session { } } + pub(crate) fn transaction_write_budget_remaining(&self) -> usize { + let used = self + .txn + .lock() + .unwrap() + .as_ref() + .map_or(0, |transaction| transaction.mem); + txn_max_bytes().saturating_sub(used) + } + pub fn database(&self) -> String { self.database.lock().unwrap().clone() } @@ -618,6 +655,19 @@ impl Session { Ok(checkpoint) } + /// Upgrade the active transaction so every subsequently scanned range is + /// validated at commit. DDL table rewrites need this even under the default + /// snapshot isolation: otherwise a concurrent insert can land in the old + /// keyspace after the rewrite's scan and survive the catalog change. + pub(crate) fn require_serializable_validation(&self) -> Result<()> { + let mut guard = self.txn.lock().unwrap(); + let transaction = guard + .as_mut() + .ok_or_else(|| Error::Query("serializable validation outside a transaction".into()))?; + transaction.serializable = true; + Ok(()) + } + pub(crate) fn release_transaction_checkpoint( &self, _checkpoint: TransactionCheckpoint, @@ -673,13 +723,34 @@ impl Session { mem: _, } = tx; + let ranges = if serializable { + coalesce_ranges(ranges) + } else { + ranges + }; + // Keys to validate = written keys, plus (serializable) read keys. // Per-table monotonic counters (`meta::…`) are excluded: they are bumped // by every write and would cause false conflicts between transactions on // the same table; real row collisions are still caught via data keys. + let range_covers = |key: &[u8]| { + serializable + && ranges.iter().any(|(start, end)| { + start.as_slice() <= key && end.as_ref().is_none_or(|end| key < end.as_slice()) + }) + }; let mut keyset: BTreeSet> = BTreeSet::new(); - keyset.extend(puts.keys().filter(|k| !is_meta(k)).cloned()); - keyset.extend(deletes.iter().filter(|k| !is_meta(k)).cloned()); + keyset.extend( + puts.keys() + .filter(|key| !is_meta(key) && !range_covers(key)) + .cloned(), + ); + keyset.extend( + deletes + .iter() + .filter(|key| !is_meta(key) && !range_covers(key)) + .cloned(), + ); keyset.extend(locked.iter().filter(|k| !is_meta(k)).cloned()); if serializable { keyset.extend(reads.iter().filter(|k| !is_meta(k)).cloned()); @@ -887,8 +958,12 @@ impl Session { // catalog epoch so cached TableDefs are refreshed. Bumping eagerly (even // for a buffered transactional write that may roll back) is safe -- it // only forces a re-read, never serves stale schema. - if puts.iter().any(|(k, _)| k.starts_with(b"catalog::")) - || deletes.iter().any(|k| k.starts_with(b"catalog::")) + if puts + .iter() + .any(|(k, _)| k.starts_with(b"catalog::") || k.starts_with(b"sys::trigger::")) + || deletes + .iter() + .any(|k| k.starts_with(b"catalog::") || k.starts_with(b"sys::trigger::")) { crate::catalog::bump_epoch(); } @@ -989,3 +1064,47 @@ where .await .map_err(|e| Error::Storage(format!("snapshot read failed: {e}")))? } + +#[cfg(test)] +mod tests { + use super::coalesce_ranges; + + fn bytes(value: &str) -> Vec { + value.as_bytes().to_vec() + } + + #[test] + fn coalesces_overlapping_and_adjacent_ranges() { + let ranges = vec![ + (bytes("m"), Some(bytes("p"))), + (bytes("a"), Some(bytes("d"))), + (bytes("c"), Some(bytes("f"))), + (bytes("f"), Some(bytes("h"))), + (bytes("n"), Some(bytes("o"))), + ]; + + assert_eq!( + coalesce_ranges(ranges), + vec![ + (bytes("a"), Some(bytes("h"))), + (bytes("m"), Some(bytes("p"))), + ] + ); + } + + #[test] + fn coalescing_preserves_nested_disjoint_and_unbounded_ranges() { + let ranges = vec![ + (bytes("z"), Some(bytes("zz"))), + (bytes("b"), Some(bytes("c"))), + (bytes("a"), Some(bytes("e"))), + (bytes("x"), None), + (bytes("y"), Some(bytes("yz"))), + ]; + + assert_eq!( + coalesce_ranges(ranges), + vec![(bytes("a"), Some(bytes("e"))), (bytes("x"), None)] + ); + } +} diff --git a/crates/elyra-engine/src/sort.rs b/crates/elyra-engine/src/sort.rs index 566861e..3386c88 100644 --- a/crates/elyra-engine/src/sort.rs +++ b/crates/elyra-engine/src/sort.rs @@ -147,6 +147,19 @@ fn pid_is_dead(_pid: u32) -> bool { struct RunReader { r: BufReader, } + +/// Best-effort path cleanup for runs moved out of `Sorter`, including when +/// reading or a downstream callback returns an error. +struct RunPaths(Vec); + +impl Drop for RunPaths { + fn drop(&mut self) { + for path in &self.0 { + let _ = std::fs::remove_file(path); + } + } +} + impl RunReader { /// Read back a spilled run from its (already-open, possibly-unlinked) file. fn from_file(mut file: File) -> Result { @@ -157,11 +170,19 @@ impl RunReader { } fn next(&mut self) -> Result, Vec)>> { let mut len = [0u8; 4]; - match self.r.read_exact(&mut len) { - Ok(()) => {} - Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None), - Err(e) => return Err(Error::Io(e)), + match self.r.read(&mut len[..1]) { + Ok(0) => return Ok(None), + Ok(1) => {} + Ok(_) => unreachable!("one-byte read returned more than one byte"), + Err(error) => return Err(Error::Io(error)), } + self.r.read_exact(&mut len[1..]).map_err(|error| { + if error.kind() == std::io::ErrorKind::UnexpectedEof { + Error::Storage("truncated sort spill record header".into()) + } else { + Error::Io(error) + } + })?; let n = u32::from_le_bytes(len) as usize; if n > elyra_core::max_frame_bytes() { return Err(Error::Storage( @@ -169,7 +190,13 @@ impl RunReader { )); } let mut buf = vec![0u8; n]; - self.r.read_exact(&mut buf)?; + self.r.read_exact(&mut buf).map_err(|error| { + if error.kind() == std::io::ErrorKind::UnexpectedEof { + Error::Storage("truncated sort spill record".into()) + } else { + Error::Io(error) + } + })?; let v = bincode::deserialize(&buf).map_err(|e| Error::Storage(e.to_string()))?; Ok(Some(v)) } @@ -320,14 +347,29 @@ impl Sorter { Ok(()) } - /// Finish sorting and return rows in order, with offset/limit applied. - pub fn finish(&mut self) -> Result>> { + /// Finish sorting, invoking `emit` once for each row in order after applying + /// offset and limit. + /// + /// Unlike [`finish`](Self::finish), this does not accumulate the result in + /// memory. This is useful when the next operator writes rows to its own + /// bounded-memory representation. + pub fn finish_with(&mut self, mut emit: F) -> Result<()> + where + F: FnMut(Vec) -> Result<()>, + { + if self.limit == Some(0) { + self.heap.clear(); + self.buffer.clear(); + self.runs.clear(); + return Ok(()); + } if self.topn { let mut ranked: Vec = self.heap.drain().collect(); ranked.sort_by(|a, b| cmp_keys(&a.keys, &b.keys, &self.asc, &self.colls)); - let rows: Vec> = ranked.into_iter().map(|r| r.row).collect(); - let start = self.offset.min(rows.len()); - return Ok(rows[start..].to_vec()); + for ranked in ranked.into_iter().skip(self.offset) { + emit(ranked.row)?; + } + return Ok(()); } if self.runs.is_empty() { @@ -336,14 +378,20 @@ impl Sorter { let colls = self.colls.clone(); let mut buffer = std::mem::take(&mut self.buffer); buffer.sort_by(|a, b| cmp_keys(&a.0, &b.0, &asc, &colls)); - let mut out: Vec> = buffer.into_iter().map(|(_, r)| r).collect(); - if self.offset > 0 { - out.drain(0..self.offset.min(out.len())); - } - if let Some(l) = self.limit { - out.truncate(l); + let rows = buffer.into_iter().map(|(_, row)| row).skip(self.offset); + match self.limit { + Some(limit) => { + for row in rows.take(limit) { + emit(row)?; + } + } + None => { + for row in rows { + emit(row)?; + } + } } - return Ok(out); + return Ok(()); } // Spill the tail, then k-way merge every run. @@ -351,7 +399,7 @@ impl Sorter { self.spill()?; } let runs = std::mem::take(&mut self.runs); - let paths: Vec = runs.iter().map(|(p, _)| p.clone()).collect(); + let _paths = RunPaths(runs.iter().map(|(path, _)| path.clone()).collect()); let mut readers: Vec = runs .into_iter() .map(|(_, f)| RunReader::from_file(f)) @@ -361,8 +409,8 @@ impl Sorter { heads.push(r.next()?); } - let mut out = Vec::new(); let mut skipped = 0usize; + let mut emitted = 0usize; loop { // Pick the smallest current head across runs. let mut best: Option = None; @@ -385,18 +433,26 @@ impl Sorter { if skipped < self.offset { skipped += 1; } else { - out.push(row); + emit(row)?; + emitted += 1; if let Some(l) = self.limit { - if out.len() >= l { + if emitted >= l { break; } } } } - for p in &paths { - let _ = std::fs::remove_file(p); - } - Ok(out) + Ok(()) + } + + /// Finish sorting and return rows in order, with offset/limit applied. + pub fn finish(&mut self) -> Result>> { + let mut rows = Vec::new(); + self.finish_with(|row| { + rows.push(row); + Ok(()) + })?; + Ok(rows) } } @@ -601,6 +657,109 @@ mod spill_tests { let rows = s.finish().unwrap(); assert_eq!(rows.len(), 25, "finish must return every pushed row"); } + + #[test] + fn finish_with_streams_sorted_rows() { + let mut sorter = Sorter::new(vec![true], vec![Collation::Ci], 1, Some(3), 2); + for i in [4, 1, 3, 0, 2] { + sorter + .push(vec![Value::Int(i)], vec![Value::Int(i)]) + .unwrap(); + } + + let mut rows = Vec::new(); + sorter + .finish_with(|row| { + rows.push(row); + Ok(()) + }) + .unwrap(); + + assert_eq!( + rows, + vec![ + vec![Value::Int(1)], + vec![Value::Int(2)], + vec![Value::Int(3)] + ] + ); + } + + #[test] + fn finish_with_propagates_callback_errors() { + let mut sorter = Sorter::new(vec![true], vec![Collation::Ci], 0, None, 2); + for i in (0..5).rev() { + sorter + .push(vec![Value::Int(i)], vec![Value::Int(i)]) + .unwrap(); + } + + let err = sorter + .finish_with(|row| { + if row == vec![Value::Int(2)] { + return Err(Error::Storage("sink failed".into())); + } + Ok(()) + }) + .unwrap_err(); + assert!(err.to_string().contains("sink failed")); + } + + #[test] + fn external_finish_with_limit_zero_emits_nothing() { + let mut sorter = Sorter::new(vec![true], vec![Collation::Bin], 0, Some(0), 1); + // `LIMIT 0` normally selects the top-N path. Force the external branch + // to cover the large-offset case where offset + limit exceeds TOPN_CAP. + sorter.topn = false; + for value in [2, 1, 0] { + sorter + .push(vec![Value::Int(value)], vec![Value::Int(value)]) + .unwrap(); + } + + let mut rows = Vec::new(); + sorter + .finish_with(|row| { + rows.push(row); + Ok(()) + }) + .unwrap(); + assert!(rows.is_empty()); + } + + #[test] + fn run_reader_rejects_partial_header_and_body() { + let partial_header_path = temp_path(); + let mut partial_header = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&partial_header_path) + .unwrap(); + partial_header.write_all(&[4, 0]).unwrap(); + partial_header.flush().unwrap(); + let mut reader = RunReader::from_file(partial_header).unwrap(); + let error = reader.next().unwrap_err(); + assert!(error + .to_string() + .contains("truncated sort spill record header")); + let _ = fs::remove_file(partial_header_path); + + let partial_body_path = temp_path(); + let mut partial_body = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&partial_body_path) + .unwrap(); + partial_body.write_all(&10u32.to_le_bytes()).unwrap(); + partial_body.write_all(b"short").unwrap(); + partial_body.flush().unwrap(); + let mut reader = RunReader::from_file(partial_body).unwrap(); + let error = reader.next().unwrap_err(); + assert!(error.to_string().contains("truncated sort spill record")); + let _ = fs::remove_file(partial_body_path); + } } #[cfg(test)] diff --git a/crates/elyra-engine/src/stream.rs b/crates/elyra-engine/src/stream.rs index 25c7634..768988e 100644 --- a/crates/elyra-engine/src/stream.rs +++ b/crates/elyra-engine/src/stream.rs @@ -5,7 +5,11 @@ //! `LIMIT`/`OFFSET`, then project — all with bounded memory. The server //! drains batches straight to the wire. -use elyra_core::{ColumnType, Result, Schema, Value}; +use std::fs::File; +use std::io::{BufReader, Read, Seek, SeekFrom}; +use std::path::PathBuf; + +use elyra_core::{ColumnType, Error, Result, Schema, Value}; use elyra_storage::Db; use sqlparser::ast::Expr; @@ -26,6 +30,14 @@ enum Source { Literal(std::vec::IntoIter>), /// Bounded-memory clustered scan over a table. Scan(Scan), + /// Length-prefixed, bincode-encoded rows owned by this stream. + Spill(Spill), +} + +struct Spill { + reader: BufReader, + path: PathBuf, + done: bool, } struct Scan { @@ -55,36 +67,50 @@ pub struct ScanSpec { /// narrow `Float`->`Int` when every non-null value is an integer, and widen /// `Int`->`Float` when any value is a float/decimal. Non-numeric columns are /// left untouched. -fn reconcile_numeric_types(schema: &mut Schema, rows: &[Vec]) { - for (i, col) in schema.columns.iter_mut().enumerate() { - if !matches!(col.ty, ColumnType::Int | ColumnType::Float) { - continue; +pub(crate) struct NumericTypeReconciler { + states: Vec<(bool, bool, bool)>, +} + +impl NumericTypeReconciler { + pub(crate) fn new(columns: usize) -> Self { + Self { + states: vec![(false, false, false); columns], } - let mut has_float = false; - let mut has_int = false; - let mut bail = false; - for r in rows { - match r.get(i) { - Some(Value::Float(_)) | Some(Value::Decimal(..)) => has_float = true, - Some(Value::Int(_)) | Some(Value::Bool(_)) => has_int = true, + } + + pub(crate) fn observe(&mut self, row: &[Value]) { + for (index, state) in self.states.iter_mut().enumerate() { + match row.get(index) { + Some(Value::Float(_)) | Some(Value::Decimal(..)) => state.0 = true, + Some(Value::Int(_)) | Some(Value::Bool(_)) => state.1 = true, Some(Value::Null) | None => {} - Some(_) => { - bail = true; - break; - } + Some(_) => state.2 = true, } } - if bail { - continue; - } - if has_float { - col.ty = ColumnType::Float; - } else if has_int { - col.ty = ColumnType::Int; + } + + pub(crate) fn reconcile(&self, schema: &mut Schema) { + for (col, &(has_float, has_int, bail)) in schema.columns.iter_mut().zip(&self.states) { + if !matches!(col.ty, ColumnType::Int | ColumnType::Float) || bail { + continue; + } + if has_float { + col.ty = ColumnType::Float; + } else if has_int { + col.ty = ColumnType::Int; + } } } } +fn reconcile_numeric_types(schema: &mut Schema, rows: &[Vec]) { + let mut reconciler = NumericTypeReconciler::new(schema.columns.len()); + for row in rows { + reconciler.observe(row); + } + reconciler.reconcile(schema); +} + impl RowStream { /// Wrap already-computed rows. The declared numeric column types are /// reconciled with the actual values so computed columns (aggregates, @@ -116,12 +142,88 @@ impl RowStream { } } + /// Build a stream over an already-open spill file. + /// + /// Rows must be bincode-encoded `Vec` values, each preceded by a + /// little-endian `u32` byte length. The stream owns the handle and removes + /// `path` when it is dropped. Callers may unlink the path before calling + /// this on platforms that support reading an unlinked open file. + pub(crate) fn spill(schema: Schema, path: PathBuf, mut file: File) -> Result { + if let Err(error) = file.seek(SeekFrom::Start(0)) { + let _ = std::fs::remove_file(&path); + return Err(Error::Io(error)); + } + Ok(Self { + schema, + src: Source::Spill(Spill { + reader: BufReader::new(file), + path, + done: false, + }), + }) + } + /// Fetch the next batch of up to `n` output rows. Empty = exhausted. pub async fn next_batch(&mut self, n: usize) -> Result>> { match &mut self.src { Source::Literal(iter) => Ok(iter.by_ref().take(n).collect()), Source::Scan(scan) => scan.next_batch(n).await, + Source::Spill(spill) => spill.next_batch(n), + } + } +} + +impl Spill { + fn next_batch(&mut self, n: usize) -> Result>> { + let mut rows = Vec::with_capacity(n.min(SCAN_CHUNK)); + while !self.done && rows.len() < n { + match self.next_row()? { + Some(row) => rows.push(row), + None => self.done = true, + } + } + Ok(rows) + } + + fn next_row(&mut self) -> Result>> { + let mut len = [0u8; 4]; + match self.reader.read(&mut len[..1]) { + Ok(0) => return Ok(None), + Ok(1) => {} + Ok(_) => unreachable!("one-byte read returned more than one byte"), + Err(error) => return Err(Error::Io(error)), + } + self.reader.read_exact(&mut len[1..]).map_err(|error| { + if error.kind() == std::io::ErrorKind::UnexpectedEof { + Error::Storage("truncated spill row frame header".into()) + } else { + Error::Io(error) + } + })?; + + let frame_len = u32::from_le_bytes(len) as usize; + if frame_len > elyra_core::max_frame_bytes() { + return Err(Error::Storage( + "spill row frame too large (corrupt?)".into(), + )); } + let mut frame = vec![0; frame_len]; + self.reader.read_exact(&mut frame).map_err(|error| { + if error.kind() == std::io::ErrorKind::UnexpectedEof { + Error::Storage("truncated spill row frame".into()) + } else { + Error::Io(error) + } + })?; + bincode::deserialize(&frame) + .map(Some) + .map_err(|error| Error::Storage(format!("invalid spill row: {error}"))) + } +} + +impl Drop for Spill { + fn drop(&mut self) { + let _ = std::fs::remove_file(&self.path); } } @@ -185,3 +287,83 @@ impl Scan { .collect() } } + +#[cfg(test)] +mod spill_tests { + use super::*; + use std::io::Write; + use std::sync::atomic::{AtomicU64, Ordering}; + + fn spill_file(frames: &[Vec]) -> (PathBuf, File) { + static SEQ: AtomicU64 = AtomicU64::new(0); + let path = std::env::temp_dir().join(format!( + "elyrasql-stream-test-{}-{}.tmp", + std::process::id(), + SEQ.fetch_add(1, Ordering::Relaxed) + )); + let mut file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(true) + .open(&path) + .unwrap(); + for frame in frames { + file.write_all(&(frame.len() as u32).to_le_bytes()).unwrap(); + file.write_all(frame).unwrap(); + } + file.flush().unwrap(); + (path, file) + } + + fn empty_schema() -> Schema { + Schema::new(vec![]) + } + + #[tokio::test] + async fn spill_stream_reads_bounded_batches_and_cleans_up() { + let expected = [ + vec![Value::Int(1)], + vec![Value::Text("two".into())], + vec![Value::Null], + ]; + let frames: Vec<_> = expected + .iter() + .map(|row| bincode::serialize(row).unwrap()) + .collect(); + let (path, file) = spill_file(&frames); + let mut stream = RowStream::spill(empty_schema(), path.clone(), file).unwrap(); + + assert_eq!(stream.next_batch(2).await.unwrap(), expected[..2]); + assert_eq!(stream.next_batch(2).await.unwrap(), expected[2..]); + assert!(stream.next_batch(2).await.unwrap().is_empty()); + assert!(path.exists()); + drop(stream); + assert!(!path.exists()); + } + + #[tokio::test] + async fn spill_stream_rejects_oversized_frames() { + let (path, mut file) = spill_file(&[]); + let oversized = u32::try_from(elyra_core::max_frame_bytes()) + .unwrap() + .checked_add(1) + .unwrap(); + file.write_all(&oversized.to_le_bytes()).unwrap(); + file.flush().unwrap(); + let mut stream = RowStream::spill(empty_schema(), path, file).unwrap(); + let err = stream.next_batch(1).await.unwrap_err(); + assert!(err.to_string().contains("too large")); + } + + #[tokio::test] + async fn spill_stream_rejects_truncated_frames() { + let (path, mut file) = spill_file(&[]); + file.write_all(&10u32.to_le_bytes()).unwrap(); + file.write_all(b"short").unwrap(); + file.flush().unwrap(); + let mut stream = RowStream::spill(empty_schema(), path, file).unwrap(); + let err = stream.next_batch(1).await.unwrap_err(); + assert!(err.to_string().contains("truncated spill row frame")); + } +} diff --git a/crates/elyra-server/tests/wire.rs b/crates/elyra-server/tests/wire.rs index 1d581af..a3f5649 100644 --- a/crates/elyra-server/tests/wire.rs +++ b/crates/elyra-server/tests/wire.rs @@ -955,6 +955,106 @@ async fn integer_width_is_enforced_and_survives_alter() { .unwrap(); } +#[tokio::test] +async fn alter_table_add_primary_key_reclusters_existing_rows_atomically() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + + c.query_drop("CREATE TABLE add_pk (tenant INT, id INT, label TEXT, INDEX label_idx(label))") + .await + .unwrap(); + c.query_drop("INSERT INTO add_pk VALUES (2, 1, 'second'), (1, 2, 'third'), (1, 1, 'first')") + .await + .unwrap(); + c.query_drop("ALTER TABLE add_pk ADD PRIMARY KEY (tenant, id)") + .await + .unwrap(); + + let rows: Vec<(i64, i64, String)> = c + .query("SELECT tenant, id, label FROM add_pk") + .await + .unwrap(); + assert_eq!( + rows, + vec![ + (1, 1, "first".into()), + (1, 2, "third".into()), + (2, 1, "second".into()), + ] + ); + let indexed: Option<(i64, i64)> = c + .query_first("SELECT tenant, id FROM add_pk WHERE label = 'second'") + .await + .unwrap(); + assert_eq!(indexed, Some((2, 1))); + assert!(c + .query_drop("INSERT INTO add_pk VALUES (1, 1, 'duplicate')") + .await + .is_err()); + + for (table, values) in [ + ("add_pk_duplicate", "(1, 'a'), (1, 'b')"), + ("add_pk_null", "(1, 'a'), (NULL, 'b')"), + ] { + c.query_drop(format!("CREATE TABLE {table} (id INT, label TEXT)")) + .await + .unwrap(); + c.query_drop(format!("INSERT INTO {table} VALUES {values}")) + .await + .unwrap(); + assert!(c + .query_drop(format!("ALTER TABLE {table} ADD PRIMARY KEY (id)")) + .await + .is_err()); + let count: Option = c + .query_first(format!("SELECT COUNT(*) FROM {table}")) + .await + .unwrap(); + assert_eq!(count, Some(2), "failed ALTER must preserve {table}"); + c.query_drop(format!("INSERT INTO {table} VALUES (2, 'still rowid')")) + .await + .unwrap(); + } +} + +#[tokio::test] +async fn alter_add_primary_key_rejects_a_concurrent_post_scan_insert() { + let srv = TestServer::start().await; + let mut ddl = srv.conn().await; + let mut writer = srv.conn().await; + + ddl.query_drop("CREATE TABLE add_pk_race (id INT, label TEXT)") + .await + .unwrap(); + ddl.query_drop("INSERT INTO add_pk_race VALUES (1, 'before')") + .await + .unwrap(); + ddl.query_drop("START TRANSACTION").await.unwrap(); + ddl.query_drop("ALTER TABLE add_pk_race ADD PRIMARY KEY (id)") + .await + .unwrap(); + + // This row is committed after the ALTER's recluster scan but before its + // transaction commits. Range validation must reject the stale rewrite. + writer + .query_drop("INSERT INTO add_pk_race VALUES (2, 'raced')") + .await + .unwrap(); + assert!(ddl.query_drop("COMMIT").await.is_err()); + + let rows: Vec<(i64, String)> = writer + .query("SELECT id, label FROM add_pk_race ORDER BY id") + .await + .unwrap(); + assert_eq!(rows, vec![(1, "before".into()), (2, "raced".into())]); + // The failed ALTER left the table in rowid mode, so a duplicate id remains + // legal and proves no PK metadata escaped the aborted transaction. + writer + .query_drop("INSERT INTO add_pk_race VALUES (1, 'still rowid')") + .await + .unwrap(); +} + /// The compact execution schema may store several MySQL declarations in the /// same physical type, but schema tooling must still see the declaration the /// user wrote and its standard width/precision metadata. @@ -2363,6 +2463,91 @@ async fn qualified_wildcard() { assert_eq!(rows, vec![(1, 1, "post".into())]); } +/// A selective predicate on the first relation should leave only a tiny driver, +/// then probe the joined table's secondary index. Residual partner predicates +/// and NULL join keys must retain ordinary INNER JOIN semantics. +#[tokio::test] +async fn selective_join_drives_secondary_index_probes() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + + c.query_drop("CREATE TABLE sj_users (id BIGINT PRIMARY KEY, name VARCHAR(32))") + .await + .unwrap(); + c.query_drop( + "CREATE TABLE sj_orders (id BIGINT PRIMARY KEY, user_id BIGINT NULL, active INT, \ + INDEX orders_user (user_id))", + ) + .await + .unwrap(); + c.query_drop("INSERT INTO sj_users VALUES (1,'one'),(2,'two'),(3,'three')") + .await + .unwrap(); + c.query_drop( + "INSERT INTO sj_orders VALUES \ + (10,1,1),(11,1,0),(12,2,1),(13,NULL,1),(14,1,1)", + ) + .await + .unwrap(); + + let rows: Vec<(String, i64)> = c + .query( + "SELECT u.name,o.id FROM sj_users u JOIN sj_orders o \ + ON u.id=o.user_id WHERE u.id=1 AND o.active=1 ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![("one".into(), 10), ("one".into(), 14)]); + + let plan: Vec = c + .query( + "EXPLAIN SELECT u.name,o.id FROM sj_users u JOIN sj_orders o \ + ON u.id=o.user_id WHERE u.id=1 AND o.active=1", + ) + .await + .unwrap(); + assert_eq!(plan.len(), 2); + assert_eq!( + plan[0].get::("table").as_deref(), + Some("sj_users") + ); + assert_eq!(plan[0].get::("key").as_deref(), Some("PRIMARY")); + assert_eq!( + plan[1].get::("table").as_deref(), + Some("sj_orders") + ); + assert_eq!( + plan[1].get::("key").as_deref(), + Some("orders_user") + ); + assert!(plan[1] + .get::("Extra") + .is_some_and(|extra| extra.contains("Indexed nested-loop join"))); + + let rows: Vec = c + .query( + "SELECT o.id FROM sj_users u JOIN sj_orders o \ + ON u.id=o.user_id WHERE u.id=3 ORDER BY o.id", + ) + .await + .unwrap(); + assert!(rows.is_empty()); + + c.query_drop("START TRANSACTION").await.unwrap(); + c.query_drop("INSERT INTO sj_orders VALUES (15,1,1)") + .await + .unwrap(); + let rows: Vec = c + .query( + "SELECT o.id FROM sj_users u JOIN sj_orders o ON u.id=o.user_id \ + WHERE u.id=1 AND o.active=1 ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![10, 14, 15]); + c.query_drop("ROLLBACK").await.unwrap(); +} + #[tokio::test] async fn relation_qualifiers_follow_mysql_case_rules() { let srv = TestServer::start().await; @@ -11699,6 +11884,155 @@ async fn wide_index_ranges_return_correct_rows() { assert_eq!(rows, want); } +#[tokio::test] +async fn composite_index_prefix_ranges_are_exact() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + c.query_drop( + "CREATE TABLE composite_ranges ( + id INT PRIMARY KEY, + tenant VARCHAR(8) COLLATE utf8mb4_bin NOT NULL, + score INT, + active INT NOT NULL, + INDEX ix_tenant_score_active (tenant, score, active) + )", + ) + .await + .unwrap(); + c.query_drop( + "INSERT INTO composite_ranges VALUES + (1,'a',10,1),(2,'a',11,1),(3,'a',12,0),(4,'a',13,1), + (5,'A',11,1),(6,'b',11,1),(7,'a',NULL,1)", + ) + .await + .unwrap(); + + // Both endpoint modes, repeated bounds (the strongest wins), binary + // collation on the equality prefix, and a residual predicate. + let ids: Vec = c + .query( + "SELECT id FROM composite_ranges + WHERE tenant = 'a' AND score >= 10 AND score > 10 + AND score <= 13 AND score < 13 AND active = 1 + ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(ids, vec![2]); + let ids: Vec = c + .query( + "SELECT id FROM composite_ranges + WHERE tenant = 'A' AND score BETWEEN 11 AND 11 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(ids, vec![5]); + + // COUNT(*) can stay entirely in the composite-index keyspace when every + // conjunct is covered. A residual or duplicate equality must retain the + // ordinary fetch-and-recheck path. + let count: i64 = c + .query_first( + "SELECT COUNT(*) FROM composite_ranges + WHERE tenant = 'a' AND score BETWEEN 10 AND 13", + ) + .await + .unwrap() + .unwrap(); + assert_eq!(count, 4); + let residual_count: i64 = c + .query_first( + "SELECT COUNT(*) FROM composite_ranges + WHERE tenant = 'a' AND score BETWEEN 10 AND 13 AND active = 1", + ) + .await + .unwrap() + .unwrap(); + assert_eq!(residual_count, 3); + + // Index maintenance and reads share the transaction snapshot. + c.query_drop("BEGIN").await.unwrap(); + c.query_drop("INSERT INTO composite_ranges VALUES (8,'a',12,1)") + .await + .unwrap(); + let ids: Vec = c + .query( + "SELECT id FROM composite_ranges + WHERE tenant = 'a' AND score >= 12 AND score <= 12 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(ids, vec![3, 8]); + c.query_drop("UPDATE composite_ranges SET score = 20 WHERE id = 2") + .await + .unwrap(); + c.query_drop("DELETE FROM composite_ranges WHERE id = 3") + .await + .unwrap(); + let ids: Vec = c + .query( + "SELECT id FROM composite_ranges + WHERE tenant = 'a' AND score BETWEEN 10 AND 13 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(ids, vec![1, 4, 8]); + c.query_drop("ROLLBACK").await.unwrap(); + + // Contradictory/repeated prefix and range predicates must never widen the + // lookup selected from the first usable composite index. + let ids: Vec = c + .query( + "SELECT id FROM composite_ranges + WHERE tenant = 'a' AND tenant = 'b' + AND score >= 10 AND score < 10 ORDER BY id", + ) + .await + .unwrap(); + assert!(ids.is_empty()); + + c.query_drop( + "CREATE TABLE composite_ci ( + id INT PRIMARY KEY, + tenant VARCHAR(8) COLLATE utf8mb4_0900_ai_ci NOT NULL, + score INT NOT NULL, + INDEX ix_ci (tenant, score) + )", + ) + .await + .unwrap(); + c.query_drop("INSERT INTO composite_ci VALUES (1,'Cafe',10),(2,'CAFÉ',11),(3,'other',10)") + .await + .unwrap(); + let ids: Vec = c + .query( + "SELECT id FROM composite_ci + WHERE tenant = 'café' AND score BETWEEN 10 AND 11 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(ids, vec![1, 2]); + + // A nullable trailing indexed column makes the composite range unsafe; + // falling back must retain rows whose index entry was intentionally omitted. + c.query_drop( + "CREATE TABLE nullable_tail ( + id INT PRIMARY KEY, a INT NOT NULL, b INT NOT NULL, tail INT, + INDEX ix_ab_tail (a, b, tail) + )", + ) + .await + .unwrap(); + c.query_drop("INSERT INTO nullable_tail VALUES (1,7,10,NULL),(2,7,11,1)") + .await + .unwrap(); + let ids: Vec = c + .query("SELECT id FROM nullable_tail WHERE a = 7 AND b >= 10 ORDER BY id") + .await + .unwrap(); + assert_eq!(ids, vec![1, 2]); +} + // `col IN (literals)` on an indexed column is served by index lookups rather than a // scan that tests membership per row. The planner may still choose a scan for a wide // list, so this pins the *results* across every shape that path has to get right -- @@ -11882,6 +12216,170 @@ async fn window_functions_are_exact() { assert_eq!(rows[0].2, 1); } +#[tokio::test] +async fn numeric_range_and_groups_window_frames_are_exact() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + c.query_drop("CREATE TABLE wf (id INT PRIMARY KEY, g INT, k INT, v INT)") + .await + .unwrap(); + c.query_drop( + "INSERT INTO wf VALUES + (1, 1, 1, 10), (2, 1, 2, 20), (3, 1, 2, 30), + (4, 1, 4, 40), (5, 1, 7, 50), (6, 2, 2, 60), + (7, 2, 5, 70), (8, 2, NULL, 80), (9, 2, NULL, 90)", + ) + .await + .unwrap(); + + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, SUM(v) OVER ( + PARTITION BY g ORDER BY k + RANGE BETWEEN 2 PRECEDING AND CURRENT ROW + ) FROM wf WHERE g = 1 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 10), (2, 60), (3, 60), (4, 90), (5, 50)]); + + let window_plan: mysql_async::Row = c + .query_first( + "EXPLAIN SELECT SUM(v) OVER (ORDER BY k RANGE BETWEEN 2 PRECEDING \ + AND CURRENT ROW) FROM wf", + ) + .await + .unwrap() + .unwrap(); + assert!(window_plan + .get::("Extra") + .is_some_and(|extra| extra.contains("Incremental window aggregate"))); + let distinct_plan: mysql_async::Row = c + .query_first("EXPLAIN SELECT DISTINCT g FROM wf") + .await + .unwrap() + .unwrap(); + assert!(distinct_plan + .get::("Extra") + .is_some_and(|extra| extra.contains("Distinct (spill-capable)"))); + + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, SUM(v) OVER ( + ORDER BY k DESC RANGE BETWEEN 2 PRECEDING AND 1 FOLLOWING + ) FROM wf WHERE g = 1 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 60), (2, 100), (3, 100), (4, 40), (5, 50)]); + + // GROUPS counts peer groups rather than physical rows and accepts a + // multi-column ordering key. + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k, g + GROUPS BETWEEN 1 PRECEDING AND 1 FOLLOWING + ) FROM wf WHERE g = 1 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 3), (2, 4), (3, 4), (4, 4), (5, 2)]); + + // A numeric RANGE offset on a NULL ordering value is its NULL peer group. + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k RANGE BETWEEN 3 PRECEDING AND 3 FOLLOWING + ) FROM wf WHERE g = 2 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(6, 2), (7, 2), (8, 2), (9, 2)]); + + c.query_drop("CREATE TABLE wf_exact (id INT PRIMARY KEY, k BIGINT)") + .await + .unwrap(); + c.query_drop( + "INSERT INTO wf_exact VALUES + (1, 9007199254740992), (2, 9007199254740993)", + ) + .await + .unwrap(); + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k RANGE BETWEEN 0 PRECEDING AND 0 FOLLOWING + ) FROM wf_exact ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 1), (2, 1)]); + + // A valid frame can lie wholly before the partition. Its upper-bound + // search must produce an empty frame rather than an out-of-range slice. + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k RANGE BETWEEN 20 PRECEDING AND 10 PRECEDING + ) FROM wf WHERE g = 1 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 0), (2, 0), (3, 0), (4, 0), (5, 0)]); + + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k GROUPS BETWEEN 4 PRECEDING AND 3 PRECEDING + ) FROM wf WHERE g = 1 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(1, 0), (2, 0), (3, 0), (4, 0), (5, 1)]); + + // DESC ordering and NULL peers exercise the opposite boundary direction. + let rows: Vec<(i64, i64)> = c + .query( + "SELECT id, COUNT(*) OVER ( + ORDER BY k DESC RANGE BETWEEN 1 PRECEDING AND CURRENT ROW + ) FROM wf WHERE g = 2 ORDER BY id", + ) + .await + .unwrap(); + assert_eq!(rows, vec![(6, 1), (7, 1), (8, 2), (9, 2)]); + + let err = c + .query_iter( + "SELECT SUM(v) OVER ( + ORDER BY k, id RANGE BETWEEN 1 PRECEDING AND CURRENT ROW + ) FROM wf", + ) + .await + .unwrap_err(); + assert!(err.to_string().contains("exactly one numeric ORDER BY")); + + let err = c + .query_iter( + "SELECT SUM(v) OVER ( + ORDER BY k RANGE BETWEEN k PRECEDING AND CURRENT ROW + ) FROM wf", + ) + .await + .unwrap_err(); + assert!(err.to_string().contains("constant expressions")); + + let err = c + .query_iter( + "SELECT SUM(v) OVER ( + ORDER BY k GROUPS BETWEEN 1.5 PRECEDING AND CURRENT ROW + ) FROM wf", + ) + .await + .unwrap_err(); + assert!(err.to_string().contains("exact non-negative integers")); +} + // Statements containing non-ASCII text must not panic the connection. The keyword // sniffers sliced the SQL by byte offset, so a multi-byte character straddling that // offset (`SELECT 'æ'='ae'` -- 'æ' spans bytes 8..10, "drop user" is 9 bytes) aborted @@ -12030,3 +12528,115 @@ async fn columns_without_a_declared_width_fall_back_to_unbounded() { assert_eq!(columns[0].column_length(), 65_535 * 4); assert_eq!(columns[1].column_length(), 65_535); } + +#[tokio::test] +async fn simple_correlated_exists_and_not_exists_preserve_sql_semantics() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + c.query_drop("CREATE TABLE corr_outer (id INT, enabled INT)") + .await + .unwrap(); + c.query_drop("CREATE TABLE corr_inner (outer_id INT)") + .await + .unwrap(); + c.query_drop("INSERT INTO corr_outer VALUES (1, 1), (2, 1), (3, 0), (NULL, 1)") + .await + .unwrap(); + c.query_drop("INSERT INTO corr_inner VALUES (1), (3), (NULL), (1)") + .await + .unwrap(); + + let exists: Vec> = c + .query( + "SELECT o.id FROM corr_outer o + WHERE o.enabled = 1 + AND EXISTS (SELECT 1 FROM corr_inner i WHERE i.outer_id = o.id) + ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(exists, [Some(1)]); + + let plan: mysql_async::Row = c + .query_first( + "EXPLAIN SELECT o.id FROM corr_outer o WHERE o.enabled = 1 \ + AND EXISTS (SELECT 1 FROM corr_inner i WHERE i.outer_id = o.id)", + ) + .await + .unwrap() + .unwrap(); + assert!(plan + .get::("Extra") + .is_some_and(|extra| extra.contains("Using semi-join membership"))); + + let not_exists: Vec> = c + .query( + "SELECT o.id FROM corr_outer o + WHERE o.enabled = 1 + AND NOT EXISTS (SELECT 1 FROM corr_inner i WHERE o.id = i.outer_id) + ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(not_exists, [None, Some(2)]); + + // Aggregate EXISTS has different semantics: COUNT returns one row even + // when the correlated WHERE matches nothing, so it must stay on the + // general correlated-subquery path rather than membership execution. + let aggregate_exists: Vec> = c + .query( + "SELECT o.id FROM corr_outer o + WHERE o.enabled = 1 + AND EXISTS (SELECT COUNT(*) FROM corr_inner i WHERE i.outer_id = o.id) + ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(aggregate_exists, [None, Some(1), Some(2)]); + + // An additional uncorrelated subquery in the residual predicate also + // deliberately falls back; direct membership only handles outer-local + // residual conjuncts. + let residual_subquery: Vec> = c + .query( + "SELECT o.id FROM corr_outer o + WHERE EXISTS (SELECT 1) + AND EXISTS (SELECT 1 FROM corr_inner i WHERE i.outer_id = o.id) + ORDER BY o.id", + ) + .await + .unwrap(); + assert_eq!(residual_subquery, [Some(1), Some(3)]); +} + +#[tokio::test] +async fn insert_trigger_cache_is_invalidated_by_trigger_ddl() { + let srv = TestServer::start().await; + let mut c = srv.conn().await; + c.query_drop("CREATE TABLE trigger_source (id INT PRIMARY KEY)") + .await + .unwrap(); + c.query_drop("CREATE TABLE trigger_audit (id INT PRIMARY KEY)") + .await + .unwrap(); + + // The first insert caches the absence of triggers. + c.query_drop("INSERT INTO trigger_source VALUES (1)") + .await + .unwrap(); + c.query_drop( + "CREATE TRIGGER source_audit AFTER INSERT ON trigger_source \ + FOR EACH ROW INSERT INTO trigger_audit VALUES (NEW.id)", + ) + .await + .unwrap(); + c.query_drop("INSERT INTO trigger_source VALUES (2)") + .await + .unwrap(); + + let audit: Vec = c + .query("SELECT id FROM trigger_audit ORDER BY id") + .await + .unwrap(); + assert_eq!(audit, [2]); +} diff --git a/crates/elyra-vector/src/lib.rs b/crates/elyra-vector/src/lib.rs index 96bda13..ec36920 100644 --- a/crates/elyra-vector/src/lib.rs +++ b/crates/elyra-vector/src/lib.rs @@ -4,8 +4,8 @@ //! MySQL-flavoured surface: `VEC_DISTANCE(a, b)` plus distance functions used //! in `ORDER BY ... LIMIT k` for approximate nearest-neighbour (ANN) search. //! -//! Milestone status: **planned** (an HNSW index backs `VECTOR` columns). -//! The distance math below is real and used for exact search / tests today. +//! An HNSW index backs indexed `VECTOR` columns for approximate nearest-neighbour +//! search. The distance math below is also used for exact search and tests. pub mod hnsw; pub use hnsw::{Hnsw, HnswParts}; diff --git a/docs/configuration.md b/docs/configuration.md index e224d2b..4d83ca1 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -29,8 +29,9 @@ variable fallback (handy for systemd and containers). | `ELYRASQL_MAX_ALLOWED_PACKET` | `67108864` (64 MiB) | Budget for one client-facing payload, mirroring MySQL's `max_allowed_packet` **and its default**. Bounds (a) one string-expanding result (`REPEAT`, `SPACE`, `LPAD`, `RPAD`), which yields `NULL` when larger — as MySQL does; and (b) parameter data streamed with `COM_STMT_SEND_LONG_DATA`, where exceeding it fails the following `EXECUTE` with error **1153** (`Got a packet bigger than 'max_allowed_packet' bytes`) and frees the buffer. | | `ELYRASQL_MAX_CONNECTIONS` | `151` | Maximum client connections served at once, mirroring MySQL's `max_connections` **and its default**. Surplus connections are refused with error **1040** (`Too many connections`) as the first packet — so clients report the real reason instead of a dropped connection — and counted in `elyrasql_connections_refused_total`. `0` disables the limit. As in MySQL, **one additional slot is reserved for administrators**: when the limit is full, one more connection is admitted and served only if it authenticates as an `Admin` account (anyone else gets 1040 after authenticating, not a misleading auth error), so an operator can still get in to diagnose a saturated server. | | `ELYRASQL_SERIALIZABLE_MAX_RANGE` | `5000000` | Max rows in a single scanned range that a `SERIALIZABLE` commit will materialize for phantom validation. A larger range aborts the commit (fail-safe against OOM) instead of buffering without limit — narrow the predicate or use a lower isolation level. | -| `ELYRASQL_IN_SUBQUERY_MAX` | `1000000` | Max rows a `WHERE col IN (SELECT ...)` may materialize into an in-memory value list. Beyond it the query errors fail-safe (rewrite as a `JOIN`/`EXISTS`) rather than buffering an unbounded list and evaluating it `O(N×M)`. | -| `ELYRASQL_DISTINCT_MAX` | `5000000` | Max distinct rows `SELECT DISTINCT` may buffer before erroring fail-safe instead of risking OOM. | +| `ELYRASQL_IN_SUBQUERY_MAX` | `1000000` | Max rows a `WHERE col IN (SELECT ...)` may materialize into an in-memory value list. Beyond it the query errors fail-safe (rewrite as a `JOIN`/`EXISTS`) rather than buffering an unbounded list and evaluating it `O(N×M)`. The same ceiling bounds the one-time key set used by simple correlated `EXISTS` decorrelation; beyond it that optimization silently falls back to nested-loop execution. | +| `ELYRASQL_DISTINCT_MAX` | `5000000` | Max distinct rows `SELECT DISTINCT` keeps in its hash fast path before switching to external sort. External-sort run size is controlled independently by `ELYRASQL_SORT_MAX_ROWS`, avoiding excessive tiny spill runs when this threshold is low. | +| `ELYRASQL_DISTINCT_MAX_BYTES` | `268435456` | Approximate byte budget for DISTINCT's in-memory keys and rows. DISTINCT spills when either this limit or `ELYRASQL_DISTINCT_MAX` is exceeded. | | `ELYRASQL_INDEX_RANGE_MAX_FRACTION` | `0.06` | Largest fraction of a table a **secondary-index** range (or `IN` list) may match before a sequential scan is used instead. An index range pays a random keyed fetch per matching row, while a scan decodes rows in storage order, so the index only wins on a small slice. The decision is made after walking the index keys but before fetching rows, so a wide range costs only a key-only walk. Primary-key ranges are unaffected — those are sequential reads. | | `ELYRASQL_JOIN_MAX_ROWS` | `10000000` | Max rows one **materialising** join may buffer (`FULL`, derived-table and multi-table `RIGHT` joins, plus any join whose output is neither aggregated nor ordered — the shapes the streaming paths do not cover. Since 1.6.0 cross *and* non-equi joins stream, so they no longer reach this ceiling; bound those with `ELYRASQL_QUERY_TIMEOUT_MS` instead). Past it the query errors instead of growing until the process is killed. Streaming joins are unaffected: they never hold the join output. | | `ELYRASQL_JOIN_MAX_BYTES` | `2147483648` (2 GiB) | Memory ceiling for rows buffered by **all** materialising joins at once. A row count is a poor proxy for memory — a wide row costs many times a narrow one — so this is the bound that actually protects the process, with the row ceilings as a cheap secondary guard. It is **approximate**: reservations are taken in blocks and the allocator retains freed memory, so peak RSS runs above the ceiling (measured ~2.3x with 1.5 KB rows). Set it well below available memory. | diff --git a/docs/limitations.md b/docs/limitations.md index f1c1fb3..1037937 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -35,8 +35,8 @@ judge fit before deploying. counters); this is a full recompute, not incremental delta maintenance. - **Named windows** are supported: `... OVER w ... WINDOW w AS (PARTITION BY ... ORDER BY ...)`, including `OVER (w ...)` inheriting a named window. -- Not yet: `RANGE`/`GROUPS` numeric value-offset frames (only - `UNBOUNDED PRECEDING .. CURRENT ROW`/`UNBOUNDED FOLLOWING` for `RANGE`), +- Numeric value-offset `RANGE` frames and peer-offset `GROUPS` frames are + supported. Temporal `RANGE` offsets are not yet supported. Other gaps include correlated subqueries combined with aggregation over a join, user-defined functions, and events. - `INSERT ... SET col = val, ...` (MySQL shorthand) is supported — it is @@ -68,7 +68,10 @@ judge fit before deploying. - Enforced: `PRIMARY KEY`, `UNIQUE`, `NOT NULL`, `CHECK`, and `FOREIGN KEY`. - Foreign keys reference a primary key or unique index; both `ON DELETE` and `ON UPDATE` `RESTRICT`/`NO ACTION`/`CASCADE`/`SET NULL` are enforced. -- Not yet: multi-level (recursive) cascades, and deferred constraint checking. +- Multi-level `ON DELETE` cascades are supported, including self-referencing + hierarchies. Delete cascades run to a fixed point with cycle detection and a + depth guard. `ON UPDATE` cascades are currently single-level, and deferred + constraint checking is not yet supported. ## Query planning @@ -192,14 +195,17 @@ judge fit before deploying. shapes are rare and the materialising path is correct. Left-deep chains of more than two tables *do* stream; a join expression the chain builder cannot analyse takes the materialising `join_select` path. -- **`WHERE col IN (SELECT ...)` and `DISTINCT` collection are in-memory** (unlike - `ORDER BY`/`GROUP BY`, which spill): the subquery's value list and the distinct - set are buffered in RAM. To stay fail-safe rather than OOM, these are **bounded** - — an `IN (SELECT ...)` over more than `ELYRASQL_IN_SUBQUERY_MAX` rows (default - 1,000,000) or a `DISTINCT` over more than `ELYRASQL_DISTINCT_MAX` rows (default - 5,000,000) errors with a clear message (rewrite `IN (SELECT)` as a `JOIN`/ - `EXISTS`). True disk-spilling for these is a future step. Correlated subqueries - execute as a nested loop (re-run per driving row, `O(N×M)`), not yet decorrelated. +- **`WHERE col IN (SELECT ...)` collection is in-memory**: the subquery's value + list is buffered in RAM. To stay fail-safe rather than OOM, an `IN (SELECT ...)` + over more than `ELYRASQL_IN_SUBQUERY_MAX` rows (default 1,000,000) errors with a + clear message (rewrite it as a `JOIN`/`EXISTS`). `SELECT DISTINCT` does spill: + it keeps up to `ELYRASQL_DISTINCT_MAX` distinct rows in its in-memory fast path + (default 5,000,000), then switches to external sorting. A narrow correlated + `EXISTS`/`NOT EXISTS` shape is decorrelated into one-time key membership: one + plain outer table, a top-level `AND` conjunct, one plain inner table, and one + type- and collation-compatible column equality. NULL keys retain SQL + semi/anti-join semantics. Other correlated shapes execute as a nested loop + (re-run per driving row, `O(N×M)`). - Uncommitted transaction writes are buffered in memory (not spilled to disk) until `COMMIT`/`ROLLBACK`. To keep this bounded, a transaction that stages more than `ELYRASQL_TXN_MAX_BYTES` (default 1 GiB) of writes has its next write @@ -441,14 +447,16 @@ judge fit before deploying. `PDO::ATTR_EMULATE_PREPARES => false` — is fixed). `describe_query` reports an exact result-column count at `PREPARE` (enable with `ELYRASQL_STMT_DESCRIBE`) for single **and** joined/multi-table SELECTs, so `SELECT *` over a join - resolves its columns. Remaining gaps: `SELECT a.*` (qualified wildcard in the - projection) and `SELECT *` over `information_schema` are not yet executed. + resolves its columns. Qualified wildcards (`SELECT a.*`) and projections over + `information_schema` are supported at both prepare and execution time. Client-side (emulated) prepared statements remain the widest-compatibility default; PyMySQL and sqlx bind client-side and are unaffected. - **`LOAD DATA INFILE`** reads a **server-side** file and bulk-inserts it (requires ADMIN, like MySQL's `FILE` privilege): `LOAD DATA INFILE '' INTO TABLE t [FIELDS TERMINATED BY '...'] [ENCLOSED BY '...'] [LINES - TERMINATED BY '...'] [IGNORE n LINES] [(cols)]`, with `\N` for NULL. Client- + TERMINATED BY '...'] [IGNORE n LINES] [(cols)]`, with `\N` for NULL. Rows are + grouped into bounded 50,000-row insert units to amortize parsing and durable + commits without allowing an individual statement to grow indefinitely. Client- side `LOAD DATA LOCAL INFILE` (streaming the file over the wire) is not supported. - Authentication offers `mysql_native_password` (default) and diff --git a/docs/mysql-compatibility.md b/docs/mysql-compatibility.md index 5fa6226..c3a726b 100644 --- a/docs/mysql-compatibility.md +++ b/docs/mysql-compatibility.md @@ -145,25 +145,25 @@ gaps: - Subqueries (`WHERE` + SELECT-list, correlated + uncorrelated, **including over joins**), derived tables, CTEs including **`WITH RECURSIVE`**, `HAVING`, window functions with **explicit `ROWS` frames** and named windows, - `GROUP BY ... WITH ROLLUP` and set operations are supported. Not yet: - `RANGE`/`GROUPS` **numeric value-offset** frames (only the `UNBOUNDED`/`CURRENT - ROW` forms of `RANGE`). + `GROUP BY ... WITH ROLLUP`, set operations, numeric value-offset `RANGE` + frames, and peer-offset `GROUPS` frames are supported. Temporal `RANGE` + offsets are not yet supported. - Views, **materialized views**, row-level triggers, and stored procedures (parameters, local and session variables, `IF`/`WHILE`/`LOOP`/`REPEAT`, cursors, condition handlers) are supported; user-defined functions and scheduled events are not. - `ALTER TABLE` supports add/drop/rename/`MODIFY`/`CHANGE` column, rename table, - `ADD INDEX`/`KEY`/`UNIQUE` (with backfill) and **`ADD FOREIGN KEY`**; - `ADD PRIMARY KEY` on an existing table must instead be declared in - `CREATE TABLE`. `SHOW CREATE TABLE` echoes `CHECK` and `FOREIGN KEY` - constraints (with their referential actions) since 1.8.0, so its output can be - replayed without losing them. + `ADD INDEX`/`KEY`/`UNIQUE` (with backfill), **`ADD PRIMARY KEY`** (with an + atomic table recluster), and **`ADD FOREIGN KEY`**. `SHOW CREATE TABLE` echoes + `CHECK` and `FOREIGN KEY` constraints (with their referential actions) since + 1.8.0, so its output can be replayed without losing them. - A broad scalar function library (string, math, date/time, JSON, `MD5`/`SHA1`/ `SHA2`, `HEX`/`UNHEX`, `FORMAT`, `FIND_IN_SET`, `FROM_UNIXTIME`, ...), statistical and bitwise aggregates (`STDDEV*`, `VAR*`, `BIT_OR`/`AND`/`XOR`), `LAST_INSERT_ID()`/`ROW_COUNT()`, `@@`system variables, and `CONVERT()`. The - MySQL shorthands `INSERT ... SET`, the `<<`/`>>`/`~` bitwise operators and - `LOAD DATA LOCAL INFILE` all work. + MySQL shorthands `INSERT ... SET` and the `<<`/`>>`/`~` bitwise operators all + work. `LOAD DATA INFILE` reads a server-side file using bounded bulk insert + units; client-streamed `LOAD DATA LOCAL INFILE` is not supported. - Vector search and `VEC_DISTANCE(...)` are ElyraSQL extensions (they mirror MySQL 9's vector direction but are not identical). - **Integer storage is always 64-bit**, but the declared width and `UNSIGNED` diff --git a/docs/sql/aggregation.md b/docs/sql/aggregation.md index 6f47bda..20a9276 100644 --- a/docs/sql/aggregation.md +++ b/docs/sql/aggregation.md @@ -147,12 +147,14 @@ AVG(v) OVER (ORDER BY id ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) SUM(v) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) ``` -`RANGE` supports the whole-partition (`UNBOUNDED PRECEDING AND UNBOUNDED -FOLLOWING`) and running (`UNBOUNDED PRECEDING AND CURRENT ROW`) forms. +`RANGE` supports numeric value offsets with one numeric `ORDER BY` expression, +including ascending and descending order. `GROUPS` supports offsets measured in +peer groups and may use multiple `ORDER BY` expressions. Temporal `RANGE` +offsets are not yet supported. !!! note - `RANGE`/`GROUPS` with numeric offsets, frame `EXCLUDE`, and named windows - are not supported. + Frame `EXCLUDE` is not supported. Named windows, including inheritance with + `OVER (window_name ...)`, are supported. ## The OLAP engine diff --git a/docs/sql/ddl.md b/docs/sql/ddl.md index c751459..1b22841 100644 --- a/docs/sql/ddl.md +++ b/docs/sql/ddl.md @@ -57,6 +57,7 @@ ALTER TABLE users CHANGE COLUMN note remark TEXT; ALTER TABLE users ALTER COLUMN status SET DEFAULT 'new'; ALTER TABLE users ALTER COLUMN status DROP DEFAULT; ALTER TABLE users ALTER COLUMN status SET NOT NULL; +ALTER TABLE users ADD PRIMARY KEY (id); ``` - **ADD COLUMN** backfills existing rows with the default (or `NULL`). Adding a @@ -68,6 +69,12 @@ ALTER TABLE users ALTER COLUMN status SET NOT NULL; renames it (`CHANGE`), and resets its options (nullability, default). The type of a primary-key column cannot be changed. - **ALTER COLUMN** sets or drops a `DEFAULT`, or toggles `NOT NULL`. +- **ADD PRIMARY KEY** validates existing values, atomically reclusters the + table on the new key, and rebuilds its secondary indexes. NULL or duplicate + key values reject the whole ALTER without changing the table. The rewrite is + buffered as one transaction and is therefore bounded by + `ELYRASQL_TXN_MAX_BYTES`; very large tables may need to be copied into a new + table instead. - Type conversions follow MySQL-style leniency (e.g. `'10'` → `10`, `99` → `'99'`). - **RENAME TABLE** re-keys the data and rebuilds index entries. diff --git a/docs/sql/queries.md b/docs/sql/queries.md index ca63acc..3c09571 100644 --- a/docs/sql/queries.md +++ b/docs/sql/queries.md @@ -33,11 +33,19 @@ ElyraSQL picks an access path automatically: |-----------|-------------|------| | `pk = ` (all key columns) | clustered point lookup | `O(log n)` | | `indexed_col = ` | secondary index | `O(log n + matches)` | +| equality on a composite-index prefix plus a range on its next column | composite secondary range scan | proportional to matches | | `col >/>=/`, `BETWEEN` on PK/indexed col | ordered range scan | proportional to matches | | `ORDER BY ASC\|DESC LIMIT n` (no filter) | clustered walk (forward/reverse), stop at `n` | `O(offset + n)` | | `ORDER BY ASC\|DESC LIMIT n` | ordered index walk (+ NULL block), stop at `n` | `O(offset + n)` | | anything else | full table scan (streaming) | `O(n)` | +`EXPLAIN SELECT ...` reports the proven access path and index name. Its `Extra` +column also identifies guaranteed indexed nested-loop joins, one-time +`EXISTS`/`NOT EXISTS` membership, incremental window aggregates, and +spill-capable `DISTINCT`. Plans outside those proven subsets are reported +conservatively rather than claiming an optimization that may fall back at +runtime. + Non-accelerated scans **stream** in bounded memory, so they never load the whole table at once. When such a scan feeds an `ORDER BY ... LIMIT k`, only the filter and sort-key columns are decoded to test a row against the top-N heap; the @@ -153,7 +161,10 @@ WHERE (SELECT COUNT(*) FROM orders o WHERE o.uid = u.id) >= 2; !!! note Correlated references must be **qualified** with the outer table's name/alias (`u.id`) so they are not confused with an inner column. This - path materialises the outer rows and runs the subquery per row. + path materialises the outer rows. A top-level `EXISTS`/`NOT EXISTS` against + one plain inner table with one type- and collation-compatible column + equality builds the inner key set once; other correlated shapes run the + subquery per outer row. ### Derived tables diff --git a/docs/sql/vector-search.md b/docs/sql/vector-search.md index 9d345d2..6ffd204 100644 --- a/docs/sql/vector-search.md +++ b/docs/sql/vector-search.md @@ -84,8 +84,9 @@ exact search. the query falls back to **exact** search, which is always correct. !!! tip - Build the index once your vectors are loaded. The first query after a - change pays a one-time rebuild cost; subsequent queries are cached. + Build the index once your vectors are loaded. The first query builds and + caches the graph; after later writes, the first query pays a one-time + reconciliation scan and subsequent queries reuse the reconciled graph. ## Hybrid search (full-text + vector, fused)