Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
220 lines (169 loc) · 8.44 KB

CONTRIBUTING.md

File metadata and controls

220 lines (169 loc) · 8.44 KB
Raw

Contributing to ElyraSQL

Thanks for your interest in improving ElyraSQL! This document explains how to set up, make changes, and get them merged.

By participating you agree to abide by our Code of Conduct, and your contributions are licensed under the project's MIT License.

Ways to contribute

  • Report bugs — open an issue with a minimal reproduction.
  • Request features — describe the problem you want solved, not just a solution.
  • Improve docs — everything under docs/ and the crate rustdoc.
  • Write code — fixes, features, tests, performance work.

Before starting significant work, please open an issue to discuss the design — it saves everyone time.

Project layout

ElyraSQL is a Cargo workspace. See docs/architecture.md for the full picture.

Crate Responsibility
elyra-core value/type model, errors, comparison, date/decimal, privileges
elyra-storage single-file ACID engine (redb), Db, MVCC Snapshot
elyra-engine parse → plan → execute, sessions/transactions, catalog, indexes
elyra-olap streaming group-aggregation kernel
elyra-vector vector metrics + HNSW index
elyra-server MySQL wire protocol, auth, TLS, prepared statements
elyra-cli the elyrasql binary

Development setup

Requires Rust 1.88+.

git clone https://github.com/kwhorne/ElyraSQL.git
cd ElyraSQL
cargo build
cargo test

Before you open a pull request

CI runs formatting, linting, build, tests, and an end-to-end smoke test. Run the same locally — they must pass:

cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings
cargo build --workspace
cargo test --workspace

The root justfile provides the same checks and common local workflows without replacing the underlying Cargo commands:

just             # list grouped recipes
just check       # formatting, Clippy, and workspace tests
just check-all   # also check docs and the isolated testbench
just run         # build and run the native release binary
just docker-run  # build and run the local Docker image

just is optional; the documented Cargo, shell, and Docker commands remain the source of truth.

SQL dump correctness stress test

testbench/sql-dump/ contains a manual differential stress tool that generates deterministic MySQL dumps and compares imports between MySQL 8.4 and ElyraSQL. It is intentionally excluded from the root Cargo workspace and CI because it is a local correctness and investigation tool rather than a product feature or a stable benchmark.

Run the default schema comparison with just stress, a generated-data run with just stress-data, or its standalone checks with just stress-check. See the testbench guide for prerequisites, profiles, artifacts, and raw commands that do not require just.

End-to-end testing

Many features are best validated against a real MySQL client. Start the server and connect with PyMySQL or the mysql CLI:

cargo run --release -p elyra-cli -- serve --listen 127.0.0.1:3307 &
python3 - <<'PY'
import pymysql
c = pymysql.connect(host="127.0.0.1", port=3307, user="root", password="", autocommit=True)
cur = c.cursor()
cur.execute("CREATE TABLE t (id BIGINT PRIMARY KEY, v TEXT)")
cur.execute("INSERT INTO t VALUES (1, 'hi')")
cur.execute("SELECT * FROM t")
print(cur.fetchall())
PY

Coding guidelines

  • Branding. Keep user-facing surfaces branded ElyraSQL. Internal engine crate/dependency names (redb, sqlparser, opensrv, …) must not leak into SQL, error messages, the CLI, or the wire handshake.
  • Correctness first. Prefer a correct, simple implementation over a fast, subtly wrong one. Add fast paths deliberately and keep a correct fallback.
  • Bounded memory. Favor streaming/batched processing over materializing whole tables where practical.
  • Be honest in docs. If a feature has limits, document them (see docs/limitations.md). Do not overclaim.
  • Tests. Add unit tests for kernels (encoding, aggregation, HNSW, …) and, for user-visible behavior, an end-to-end check against the MySQL protocol.
  • Docs. Update docs/ for any user-visible change.

Commit and PR conventions

  • Write focused commits with clear, imperative messages (e.g. Add range index scans).
  • Explain the why in the body when it isn't obvious.
  • Keep PRs scoped to one logical change; open separate PRs for unrelated work.
  • Fill out the pull request template and link the issue it addresses.
  • Rebase on main and ensure CI is green.

Cutting a release

The release workflow validates that the tag, Cargo.toml and the CHANGELOG agree, so a mismatch fails before anything is published. Three things it does not check, and which have each been forgotten at least once:

  1. SERVER_VERSION in crates/elyra-core/src/lib.rs — the string clients see from SELECT VERSION(). Nothing fails if it lags; the server just misreports itself.

  2. testbench/sql-dump/Cargo.lock. The testbench is a separate Cargo workspace, so its lockfile records the workspace crate versions and goes stale on every version bump. scripts/run.sh invokes cargo run --locked, so a stale lock means the nightly SQL dump differential fails before it does any work. Refresh it with:

    (cd testbench/sql-dump && cargo check --offline)

    This went unnoticed from 1.9.1 to 1.9.4 because nothing ran the harness.

  3. Version strings in prose: README.md (the stable-release line, the SELECT VERSION() example and the Docker tags), docs/installation.md, docs/deployment.md, docs/mysql-compatibility.md.

If a release tightens validation or changes anything a working deployment can depend on, say so in an upgrade block in docs/installation.md. The version number alone is not a warning.

Reporting security issues

Please do not open public issues for vulnerabilities. See SECURITY.md.

Profile-guided optimization (PGO)

The release workflow includes a fail-safe PGO step that generates a merged.profdata training profile and feeds it to the linker. If anything fails — missing llvm-tools-preview, training data not generated, benchmark scripts changed — the step fails visibly and the run is annotated with a warning, but the build still falls back to [profile.dist] without PGO.

A release built without a valid profile is a supported release. Do not let a profiling hiccup block a release.

Maintaining the training workload

The profile is generated by scripts/pgo-build.sh --profile-only using:

  • Pre-generated SQL dumps from scripts/generate-training-sql.py
  • The OLTP / OLAP / late-materialisation benchmark suite under bench/

Profile quality tracks training size. The defaults (50k-row SQL dumps, BENCH_ROWS=50000, OLAP_ROWS=500000, LATEMAT_ROWS=100000) are the smallest workload at which PGO has been measured to beat a plain dist build; a thinner profile can make the binary slower on unrepresented paths, and nothing detects that at build time. Do not shrink them to save CI minutes.

When something breaks:

  1. If scripts/pgo-build.sh exits with "no training SQL dumps found", run python3 scripts/generate-training-sql.py --all to regenerate them.
  2. If a benchmark script signature changed (new flags, renamed arguments), update the corresponding python3 bench/*.py invocation in the script.
  3. If llvm-profdata is not found, install the LLVM tools: rustup component add llvm-tools-preview.
  4. If the profile format changes across Rust toolchain upgrades, delete target/pgo/ and re-run the pipeline.

Verifying PGO locally

# Generate training SQL data
python3 scripts/generate-training-sql.py --all

# Full pipeline: instrument → train → merge → rebuild
./scripts/pgo-build.sh

# Benchmark the rebuilt binary against a fresh data directory (not the
# training directory, to avoid warm-cache bias):
mkdir -p /tmp/pgo-bench
elyrasql serve --data /tmp/pgo-bench/bench.edb --listen 127.0.0.1:3309 --password "" &
python3 bench/benchmark.py --port 3309 --rows 50000 --password ""

For a cold A/B comparison of a baseline vs PGO binary (fresh server and fresh data directory per run, data deleted afterwards), use scripts/bench-cold.py.

License

By contributing, you agree that your contributions are licensed under the MIT License, and that you have the right to submit them.