From 7e40a7b21ad0cd38b167ec0da61f32cf00465c0a Mon Sep 17 00:00:00 2001 From: Christoph Dyllick-Brenzinger Date: Wed, 4 Nov 2026 12:06:03 +0100 Subject: [PATCH] Document releases, verified SQL behaviour and model testing CLAUDE.md gets the release steps, the SeaTable SQL behaviour the tool texts rely on, the error hints and truncation notice, the reach of the server instructions, and the container restrictions. docs/testing-with-models.md holds the test data and prompts used to check how real models use the SQL tools, so the series can be repeated after changes to tool descriptions or instructions. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 33 ++++++++++++++++++ README.md | 2 ++ docs/testing-with-models.md | 68 +++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+) create mode 100644 docs/testing-with-models.md diff --git a/CLAUDE.md b/CLAUDE.md index 3520103..ef8c1d6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -100,11 +100,44 @@ That guard exists because the opposite behaviour hid a bug for five months (fixe Every failure is logged as `SQL statement failed` with `reasons` and `sql_shape` and counted in `mcp_sql_failures_total`. `sqlShape()` removes all customer content — literals become `?`/`0`, table and column names become `x` — so the log shows which constructs models get wrong without storing data. Never log the raw statement or its parameters. How to read and act on the log: README, "SQL failure log". +`runSql()` also adds hints to SeaTable's errors, in `ERROR_HINTS`: `no such column/table` points to `get_schema` (reason `unknown_name`), `failed to convert value` and `type mismatch` explain select options and the date format (reason `invalid_value`). In production the failures were guessed names, not dialect problems, and models repeated them. + +`truncationNotice()` adds a `notice` field to `query_sql` results that may be cut, and `list_rows` does the same when `has_more` is set. A `LIMIT` below 100 is treated as a deliberate top-N and stays quiet. + +SeaTable SQL behaviour verified against 7.0, which the tool texts rely on: + +- A `SELECT` without `LIMIT` returns 100 rows, also with `GROUP BY`, and does not say it cut. `LIMIT` goes up to 10,000 and `OFFSET` works (`LIMIT n OFFSET m` and `LIMIT m, n`). +- A link column can't be a join condition: `= other.col` and `= other._id` return nothing, `HAS ANY OF (other.col)` is an error. `GROUP BY` on it works, `` `Link` = 'display value' `` filters, and the link column on the other table supports `IS NULL`. +- `ORDER BY` accepts columns outside the `SELECT` list. +- A deleted API token answers 403 on a self-hosted 7.0 and 404 on SeaTable Cloud. `TokenValidator.check()` treats both, and 401, as revoked. + +To see how real models use the tools, run the series in `docs/testing-with-models.md`. + +### Server instructions and tool annotations + +The `instructions` in `src/mcp/server.ts` reach every client, and the SeaTable AI chat plugin (`seatable-plugin-ai-chat`) embeds them in its system prompt instead of keeping its own SQL rules. A change there takes effect for all clients on the next deployment, without a plugin release. If a rule moves here from a client, deploy the server first. + +`tests/server.spec.ts` pins every tool as read-only, additive or destructive. A tool that can overwrite or remove existing data must be destructive; directory reviewers call the tools and compare against the hints. + +### Container + +The image's `CMD` runs `node --permission --allow-fs-read=/app`. Writing files, reading outside `/app`, child processes, worker threads and native addons fail with `ERR_ACCESS_DENIED`, so a new feature that needs one of these has to change the flags deliberately. The image has no `wget`, `nc`, npm or apk; health checks use `node`. See README, "Container hardening". + ### Schema Utilities - `src/schema/map.ts` — converts SeaTable metadata to `GenericSchema` format - `src/schema/validate.ts` — validates row data against schema before writes +## Releases + +The workflow runs on tags. `testing-vX.Y.Z` only builds the image `testing-X.Y.Z`. `release-vX.Y.Z` also publishes `:latest`, npm (irreversible) and the MCP Registry. + +1. Bump the version in the PR (`npm version X.Y.Z --no-git-tag-version`). The npm job fails if the tag and `package.json` differ. +2. After the merge, build the image and scan it locally with the workflow's Trivy parameters. The workflow pushes the image **before** it scans, so a red scan has already published. +3. Set an annotated tag `release-vX.Y.Z` on `main`. Its message is the changelog: name breaking changes and what deployments with their own compose file have to do. + +The commit messages of this public repo should not contain table or column names from customer logs. + ## Known Limitations - HTTP sessions are in-memory and do not survive restarts. diff --git a/README.md b/README.md index 80ee328..94fa4e9 100644 --- a/README.md +++ b/README.md @@ -486,6 +486,8 @@ node scripts/mcp-call.cjs list_tables '{}' node scripts/mcp-call.cjs list_rows '{"table": "Tasks", "page_size": 5}' ``` +To check how real models use the SQL tools, see [Testing the SQL tools with a model](docs/testing-with-models.md). + ## Troubleshooting | Issue | Solution | diff --git a/docs/testing-with-models.md b/docs/testing-with-models.md new file mode 100644 index 0000000..efc43dd --- /dev/null +++ b/docs/testing-with-models.md @@ -0,0 +1,68 @@ +# Testing the SQL tools with a model + +Unit tests can't show whether a model actually uses the tools well. Run this series after changing tool descriptions, the server instructions or the SQL checks, and ideally with a small model too: a strong model hides gaps in the instructions that a weaker one runs into. + +Check two things for each prompt: whether the answer is right, and which calls it took. A wrong answer can come without any error, for example from a result cut at 100 rows. + +## Test data + +Create a base with two tables. Import the CSV files, then set the column types. + +**Kunden:** `Umsatz` number, `Status` single select (`aktiv`, `inaktiv`), `Kunde seit` date. + +```csv +Name,Ort,Umsatz,Status,Kunde seit +Müller GmbH,Mainz,12000,aktiv,2021-03-01 +Schmidt AG,Wiesbaden,4500,inaktiv,2019-07-15 +Weber KG,Mainz,8800,aktiv,2023-11-20 +Fischer OHG,Frankfurt,300,aktiv,2024-01-05 +Becker & Söhne,Frankfurt,15200,aktiv,2020-05-12 +Hoffmann IT,Darmstadt,2100,aktiv,2022-09-30 +Klein Logistik,Mainz,6700,inaktiv,2018-02-14 +Wagner Bau,Wiesbaden,950,aktiv,2025-04-01 +``` + +**Bestellungen:** `Kunde` link to `Kunden`, `Betrag` number, `Datum` date. Use a real link column: it is what makes prompts 1 and 2 hard. SeaTable then also creates a link column `Bestellungen` in `Kunden`. + +```csv +Nr,Kunde,Betrag,Datum +B-1001,Müller GmbH,500,2026-07-03 +B-1002,Müller GmbH,700,2026-07-18 +B-1003,Weber KG,250,2026-07-22 +B-1004,Becker & Söhne,3200,2026-08-05 +B-1005,Becker & Söhne,1800,2026-08-19 +B-1006,Hoffmann IT,420,2026-08-27 +B-1007,Müller GmbH,950,2026-09-02 +B-1008,Weber KG,1300,2026-09-15 +B-1009,Becker & Söhne,640,2026-10-08 +B-1010,Hoffmann IT,2100,2026-10-21 +``` + +## Prompts + +The prompts are in German on purpose, to match the data. Run prompt 7 last, since it changes data, and set both customers back to `aktiv` afterwards. + +| # | Prompt | Tempts the model to | Good path | Right answer | +| --- | ---------------------------------------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | +| 1 | Welche Kunden haben noch keine Bestellung? | subquery, `LEFT JOIN` | one `query_sql`: ``WHERE `Bestellungen` IS NULL`` | Schmidt AG, Fischer OHG, Klein Logistik, Wagner Bau | +| 2 | Zeig mir pro Kunde die Summe aller Bestellungen und den Ort. | `JOIN`, a join on the link column (returns nothing) | `` GROUP BY `Kunde` ``, then a second query for the cities | Becker & Söhne 5.640 (Frankfurt), Hoffmann IT 2.520 (Darmstadt), Müller GmbH 2.150 (Mainz), Weber KG 1.550 (Mainz) | +| 3 | Welcher Kunde hat die größte Einzelbestellung? | `WHERE x = (SELECT MAX…)` | `ORDER BY … DESC LIMIT 1` | Becker & Söhne, B-1004, 3.200 | +| 4 | Teile die Kunden nach Umsatz ein: ab 10.000 groß, ab 2.000 mittel, darunter klein. | `CASE WHEN` | `ifs()`, or read the rows and classify them | groß: Müller, Becker · mittel: Weber, Klein, Schmidt, Hoffmann · klein: Wagner, Fischer | +| 5 | Zeig die ersten drei Buchstaben jedes Namens zusammen mit dem Ort in einer Spalte. | `SUBSTRING`, `CONCAT` | `concatenate(mid(…), …)` | e.g. "Mül Mainz", "Bec Frankfurt" | +| 6 | Wie viele Bestellungen gab es pro Monat? | `DATE_FORMAT`, `EXTRACT` | `month()`/`year()` with `GROUP BY` | July 3, August 3, September 2, October 2 | +| 7 | Setze alle Kunden aus Frankfurt auf inaktiv. | sending the `UPDATE` to `query_sql` | `execute_sql` with `UPDATE … WHERE` | 2 rows: Fischer OHG, Becker & Söhne | + +## What to watch + +Follow the server log while testing: + +```bash +docker logs -f --since 1m seatable-mcp 2>&1 | grep --line-buffered -E '"tool":|SQL statement failed' \ + | jq -rc '[.time[11:19], .tool, .msg, (.reasons // empty | join(",")), (.sql_shape // empty)] | join(" ")' +``` + +- **Ideally no rejection at all**: the model follows the description and asks simple queries. +- **After a failed call**, the next one should restructure the statement, not repeat it with small changes. +- **A row count of exactly 100** without a `notice` in the result would be a bug in the truncation notice. + +For the cause of a failure, see README, "SQL failure log".