[pull] main from MaterializeInc:main

.agents/skills/mz-debug-ci/bk-fallback.md

@@ -0,0 +1,67 @@
+# bk fallback for mz-debug-ci
+
+Command recipes for sessions without the Buildkite MCP. The semantics (job
+states, annotation contents, triage rules) live in SKILL.md; only the `bk`
+mechanics live here.
+
+`bk` writes warnings to stderr, including a benign `BUILDKITE_API_TOKEN is
+overriding the credential stored for this organization` on healthy setups.
+Pipe `bk` output with `2>/dev/null`, never `2>&1`, or the noise corrupts the
+JSON.
+
+## Listing a build's failed jobs
+
+```bash
+bk api "/pipelines/<PIPELINE>/builds/<BUILD_NUMBER>" --no-pager 2>/dev/null \
+  | jq -r '.jobs[] | select(.state=="failed" or .state=="timed_out") | select(.soft_failed != true) | [.state, .id, .name] | @tsv'
+```
+
+The same response carries the build's `.commit` and `.branch`. Add
+`?include_retried_jobs=true` to see the earlier attempts of retried jobs
+(exposes `.retried` and `.retry_source`).
+
+## Annotations
+
+```bash
+bk api /pipelines/<PIPELINE>/builds/<BUILD_NUMBER>/annotations --no-pager 2>/dev/null \
+  | jq -r '.[] | select(.style=="error") | "=== \(.context)\n\(.body_html)"' \
+  | sed -e 's/<[^>]*>//g' | grep -v '^$'
+```
+
+## Job logs
+
+```bash
+bk job log <JOB_ID> -p <PIPELINE> -b <BUILD_NUMBER> --no-timestamps --no-pager 2>/dev/null | tail -100
+```
+
+For large logs, grep for the error instead of tailing. Buildkite logs are
+full of ANSI escapes (colors, plus cursor-movement and OSC sequences from
+docker pulls) and embedded carriage returns that leave doubled blank lines,
+halving grep's `-A`/`-B` reach. Normalize first:
+
+```bash
+bk job log ... 2>/dev/null | tr '\r' '\n' \
+  | sed -e 's/\x1b\[[0-9;]*[a-zA-Z]//g' -e 's/\x1b\][^\x07]*\x07//g' | grep -v '^$' \
+  | grep -B2 -A5 'error\|FAIL\|panicked'
+```
+
+## Artifacts
+
+```bash
+# All artifacts of a build (each entry has id, job_id, path)
+bk artifacts list <BUILD_NUMBER> -p <PIPELINE> --no-pager 2>/dev/null
+# Artifacts of a single job
+bk artifacts list <BUILD_NUMBER> -p <PIPELINE> --job-uuid <JOB_ID> --no-pager 2>/dev/null
+# Download one artifact into the current directory
+bk artifacts download <ARTIFACT_ID> --build <BUILD_NUMBER> -p <PIPELINE>
+```
+
+NOTE: if `bk` rejects one of these flags as unknown, the installed `bk`
+predates the April 2026 artifacts rework. Check
+`bk artifacts <subcommand> --help` for the local spelling, and recommend to
+the user that they upgrade `bk`.
+
+Do not use `bk api` for artifacts. The plural
+`/pipelines/<PIPELINE>/builds/<BUILD_NUMBER>/artifacts` endpoint works but
+silently returns only the first 30 entries, and its `.../download` endpoint
+fails to parse.

.agents/skills/mz-debug-ci/rare-failures.md

@@ -0,0 +1,51 @@
+# Rare failure classes for mz-debug-ci
+
+Triage patterns for failure classes that come up rarely. SKILL.md's Step 4
+points here by keyword.
+
+## Feature benchmark regressions
+
+A `New regression against <version>` annotation shows one comparison table,
+but the harness re-runs every flagged scenario and annotates only the last
+run: grep the console log for `Run [0-9] with scenarios` and `!!YES!!` to
+see whether the scenario survived every re-run. Even surviving all re-runs
+does not prove a code cause, since agent-level speed offsets are stable
+within a job, and these scenarios trip on noise regularly, so check the
+base rate with `bin/ci-failures '<ScenarioName>'`. The decisive check is
+reproducing locally: check out the exact commit the build ran and run the
+reproducer command from the annotation. mzbuild pulls the images CI already
+published for that commit, so nothing needs to build. Pass `--release` for
+nightly and release-qualification builds (they build with `CI_LTO=1`); PR
+test builds and flag-less local runs both use the optimized profile, so
+no flag there.
+
+## cargo-fuzz crashes
+
+The fuzz target's own panic text is not in the console log; it is in
+`<target>.log` inside the `fuzz-logs.tar.zst` build artifact, and the
+annotation names the file. For known-vs-new use
+`bin/ci-failures '<target name>'`; cargo-fuzz runs only in release
+qualification, so that is its entire history.
+
+## Limits test failures
+
+For red `Product limits` / `Instance size limits` jobs, the mz-limits-test
+skill explains the framework, its typical failure modes, and
+single-scenario local reruns.
+
+## Miri failures
+
+`:rust: Miri test (full)` fails as either `timed_out` (a test too slow under
+Miri's interpreter) or `failed` with `unsupported operation` (code Miri
+cannot execute, e.g. a foreign function). Neither is a product bug, and the
+standard reaction is not to make the code Miri-compatible: add
+`#[cfg_attr(miri, ignore)]` with a short reason to the offending test, as
+about a thousand existing tests do.
+
+## `ImagesNotPublicError`
+
+A PR adding a new publishable mzbuild image fails `:rust: Build x86_64`
+until the image's DockerHub/GHCR repos are made public, a process step
+described by the `images-not-public` annotation. The check runs only on the
+x86_64 build, so aarch64 passing while x86_64 fails is expected, not an
+arch bug.

.agents/skills/mz-test/SKILL.md

@@ -198,6 +198,9 @@ If a flake cannot be reproduced locally, suggest the user trigger CI runs manual
 Removing the "Target Branch" generates a random identifier, allowing parallel runs.
 10-20 runs to reproduce a flake is fine.
 
+To check whether a failure is already known and chronic in CI, see the
+mz-debug-ci skill's `bin/ci-failures` search.
+
 ## Additional logging for flaky tests
 
 Add logging through `log_filter` in the test's `mzcompose.py`:

ci/nightly/pipeline.template.yml

@@ -168,6 +168,7 @@ steps:
           args: [--miri-full]
     agents:
       queue: hetzner-aarch64-16cpu-32gb
+    skip: "Has been creating more toil than use"
     sanitizer: skip
 
   - group: Benchmarks

ci/release-qualification/pipeline.template.yml

@@ -278,12 +278,9 @@ steps:
           composition: cargo-fuzz
           ci_builder: nightly
           args:
-            - --profile=fruitful
+            - --profile=all
             - --max-seconds=86400
             - --wall-budget=84600
-            # Step hard-times out at 1440min (86400s). --wall-budget ends fuzzing
-            # at 84600s, leaving 1800s; cap minimize at 1200s so the corpus
-            # upload has ~600s of headroom before the kill.
             - --minimize-timeout=1200
             - --corpus-sync
     artifact_paths:

misc/python/materialize/cli/fmt.py

@@ -18,6 +18,12 @@
 from materialize import MZ_ROOT
 from materialize.parallel_task import TaskSpec, run_parallel
 
+# Rust sources live in more than one workspace, and `cargo metadata` only ever
+# reports the one it is pointed at. The `src/*/fuzz` cargo-fuzz crates attach to
+# the `test/cargo-fuzz` workspace, which the root workspace does not include, so
+# without a second invocation here they go unformatted entirely.
+RUST_MANIFESTS = ["Cargo.toml", "test/cargo-fuzz/Cargo.toml"]
+
 
 def main() -> int:
     parser = argparse.ArgumentParser(prog="fmt")
@@ -45,40 +51,55 @@ def _rustfmt_fn(*, check: bool):
     def run() -> tuple[bool, str]:
         ncpus = os.cpu_count() or 8
 
-        result = subprocess.run(
-            ["cargo", "metadata", "--no-deps", "--format-version=1"],
-            capture_output=True,
-            text=True,
-        )
-        if result.returncode != 0:
-            return False, result.stderr.strip()
-
-        meta = json.loads(result.stdout)
         kinds = {"lib", "bin", "bench", "test", "example", "proc-macro", "custom-build"}
-        paths = [
-            t["src_path"]
-            for pkg in meta["packages"]
-            for t in pkg["targets"]
-            if kinds & set(t["kind"])
-        ]
-        if not paths:
+        # Keyed by edition: `gen` is an identifier in 2021 but a reserved keyword
+        # in 2024, so rustfmt cannot even parse a file at the wrong edition.
+        paths_by_edition: dict[str, list[str]] = {}
+        for manifest in RUST_MANIFESTS:
+            result = subprocess.run(
+                [
+                    "cargo",
+                    "metadata",
+                    "--no-deps",
+                    "--format-version=1",
+                    f"--manifest-path={manifest}",
+                ],
+                capture_output=True,
+                text=True,
+            )
+            if result.returncode != 0:
+                return False, result.stderr.strip()
+
+            meta = json.loads(result.stdout)
+            for pkg in meta["packages"]:
+                for t in pkg["targets"]:
+                    if kinds & set(t["kind"]):
+                        paths_by_edition.setdefault(pkg["edition"], []).append(
+                            t["src_path"]
+                        )
+        if not paths_by_edition:
             return True, ""
 
         # Split into batches and run rustfmt in parallel.
-        batch_size = math.ceil(len(paths) / ncpus)
-        batches = [paths[i : i + batch_size] for i in range(0, len(paths), batch_size)]
+        batches = []
+        for edition, paths in paths_by_edition.items():
+            batch_size = math.ceil(len(paths) / ncpus)
+            batches += [
+                (edition, paths[i : i + batch_size])
+                for i in range(0, len(paths), batch_size)
+            ]
 
         cmd_base = ["rustfmt", "--config", "error_on_line_overflow=true"]
         if check:
             cmd_base.append("--check")
 
         procs = [
             subprocess.Popen(
-                cmd_base + batch,
+                cmd_base + [f"--edition={edition}"] + batch,
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT,
             )
-            for batch in batches
+            for edition, batch in batches
         ]
 
         all_output = []

misc/python/materialize/version_ancestor_overrides.py

@@ -28,6 +28,12 @@ def get_ancestor_overrides_for_performance_regressions(
 
     min_ancestor_mz_version_per_commit = dict()
 
+    if scenario_class_name == "MySqlInitialLoadMultiWorkerSampled":
+        # PR#38094 (storage: Fast approximate snapshot partitioning) increased wallclock by ~30%
+        min_ancestor_mz_version_per_commit[
+            "de8eac0d1683ffd788dfba6edaa88aef7f0f1740"
+        ] = MzVersion.parse_mz("v26.39.0")
+
     if scenario_class_name == "FastPathFilterNoIndex":
         # PR#38151 (Sql-150: Temporary objects to the catalog) increased wallclock by ~10%
         min_ancestor_mz_version_per_commit[

rustfmt.toml

@@ -1,3 +1,8 @@
 edition = "2024"
+# Formatting style is repo-wide, independent of the edition a crate parses as.
+# The `src/*/fuzz` crates are still edition 2021, and `bin/fmt` overrides
+# `edition` per package so they parse at all. Without this pin they would also
+# fall back to the 2021 *style*, which sorts imports differently.
+style_edition = "2024"
 tab_spaces = 4
 merge_derives = false

src/avro/fuzz/Cargo.toml

@@ -20,6 +20,10 @@ cargo-fuzz = true
 [dependencies]
 libfuzzer-sys = "0.4"
 mz-avro = { path = ".." }
+# Version must match the root `Cargo.toml`'s. `avro_schema_parse` needs a
+# `serde_json::Value` to reach `Schema::parse`, and that value has to be the same
+# type `mz-avro` was compiled against.
+serde_json = "1.0.150"
 
 [[bin]]
 name = "reader_decode"

src/avro/fuzz/corpus.dict

@@ -1,10 +1,16 @@
-# libFuzzer dictionary for the Avro reader_decode target.
+# libFuzzer dictionary for the Avro fuzz targets, resolved crate-wide by
+# `dict_for` in test/cargo-fuzz/mzcompose.py.
 #
 # An Avro object-container file starts with the 4-byte magic "Obj\x01" followed
 # by a map of metadata (notably "avro.schema" and "avro.codec") and a 16-byte
-# sync marker. Without the magic the decoder bails immediately, so the most
-# valuable token by far is the magic itself; the rest let the mutator build
-# plausible headers and schemas.
+# sync marker. Without the magic the decoder bails immediately, so the magic and
+# the schema JSON tokens let the mutator build plausible headers and schemas.
+#
+# These tokens pay off for the targets that consume their input as *bytes*:
+# `avro_schema_parse`, and `reader_decode`'s raw-bytes branch. `reader_decode`'s
+# container-building branch and `schema_resolve` instead read their input as an
+# `arbitrary::Unstructured` recipe of generator choices, where a token is just
+# more choice bytes and carries no structural signal.
 
 # Object-container magic.
 "Obj\x01"