diff --git a/.changeset/proud-pumas-tap.md b/.changeset/proud-pumas-tap.md new file mode 100644 index 0000000..9ad664b --- /dev/null +++ b/.changeset/proud-pumas-tap.md @@ -0,0 +1,55 @@ +--- +"@cssert/cli": minor +--- + +Reduce the friction of adopting cssert in a real project. Four defaults change; +each has an opt-out, and `README.md` has an "Upgrading from 0.1" table. + +**Breaking: relative paths in a config file now resolve against the config +file's directory**, not the working directory, the way eslint/vitest/tsc do. +The same config gives the same result from any directory, which is what makes +cssert usable when `node_modules` lives in a subdirectory +(`cd frontend && cssert check --config ../cssert.config.mjs` used to match +nothing). Globs passed as flags stay relative to cwd. New `root` (config, +relative to the config file) and `--root` (CLI, relative to cwd) override the +base; `resolveFrom: "cwd"` restores the old behaviour. + +**Breaking: `--format github` emits one annotation per class**, anchored at the +first occurrence with the rest counted in the message +(`… (and 65 other places).`). GitHub caps the annotations it displays per check +run, so a single class used in 85 places used to hide every other finding — +worst on the first run, when the report is longest. `--annotate-occurrences` +restores per-occurrence annotations. Human and SARIF output are unchanged. + +**Breaking: a baseline file that does not exist no longer exits 2.** cssert +notes it on stderr and continues without suppression, so you can write the +config before freezing anything. `--require-baseline` makes the absence fatal; +`--no-baseline` (or `--baseline ""`, which used to fail with EISDIR) ignores an +existing baseline for one run. + +**Breaking: `baseline create` freezes only the kinds that fail the check** +(`--kind failing`, the new default: `missing`, plus `dynamic-suspect` under +`--fail-on-dynamic`). Freezing dynamic-suspect findings keyed by whole template +expressions made baselines rot on any edit to the expression. `--kind all` +restores the old behaviour; `--kind missing|dynamic-suspect` selects explicitly. + +Added: + +- `--min-documents` / `--min-stylesheets` (and `minDocuments` / + `minStylesheets`) fail the run when fewer inputs arrived than expected, so a + CI artefact that failed to download cannot pass as a green check. Reported in + every format through the new `Report.errors`. +- The summary counts documents that still contain unresolved class expressions, + so a green run says how much of the input it could not resolve. +- JSON findings carry `sources`: the input globs the finding was seen under. + A finding listed only under your template glob is a page nobody rendered. + `audit()` accepts `source` on each document. +- `hooks` / `--hook` for classes that are supposed to have no styles — + `allow` semantics, separate key so the config records the reason. +- `baseline create|prune --dry-run`. +- `--version` names the package (`cssert 0.2.0 (@cssert/cli)`). + +Docs: "What a green run does and does not prove" and a baseline operations +guide in the README; the Django recipe gains a pytest-based rendering recipe, +a note on rendering both branches of every `{% if %}`, and a complete +three-job GitHub Actions workflow. diff --git a/README.md b/README.md index d05f52d..04d6553 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,23 @@ HTML, and reports the difference. It has no dependency on any build tool, framework internals or browser, so it works the same for Tailwind v3, Tailwind v4, UnoCSS, PurgeCSS output and hand-written CSS. +### What this finds in practice + +Reported from a Django + Tailwind v4 + daisyUI codebase (35 templates, 27 +rendered pages) that adopted cssert in CI: + +- **The first run found 7 dead class names used in about 200 places** — leftovers + from a daisyUI v4 → v5 upgrade that nobody had noticed, because every page + still rendered and only looked slightly wrong. +- **Two past incidents were reproduced and both failed the check**: a + hand-written `.mention` rule deleted from `style.css`, and a Prettier run that + reflowed a template until `tooltip-right menu-active` became + `tooltip-rightmenu-active`. +- **The dev-server trap is caught by the same mechanism**: add a new template + while a Tailwind dev server is running, and the classes that appear for the + first time in that file are never scanned. The page works for whoever has the + old stylesheet cached and breaks in production. + ## Quick start ```sh @@ -53,6 +70,34 @@ Without installing, run `npx @cssert/cli check ...`. Exit codes: `0` no violations, `1` violations, `2` usage or config error, `3` internal error. +## What a green run does and does not prove + +`✓ no missing classes` is a statement about the files cssert was given, and +only about *existence*. Quote this section in your own CI design notes. + +**A green run proves**, for every document you passed in: every literal class +in a scanned attribute is defined by at least one selector in the stylesheets +you passed in. + +**A green run does not prove:** + +- **That the rule actually applies.** A class can exist and still lose to + another selector's specificity, sit behind a media query that never matches, + or declare the wrong value. cssert compares names, not computed style. +- **That the class is on the right element.** A `class` attribute you forgot + to write is not a missing class; it is missing markup, and nothing here sees + it. +- **That the pages you did not render are fine.** A template that no scanned + document exercises is simply not part of the result. The summary reports the + document count for this reason, `--min-documents` turns a truncated input set + into a failure, and the count of documents that still contain unresolved + class expressions says how much of the input could not be resolved. +- **That both branches were taken.** Rendering `{% if expired %}` once only + covers the branch your fixture data happened to hit. + +For "this class exists *and* resolves to this value", use +[`expectClass`](#explicit-assertions). + ## Server-side rendering: check rendered pages, not templates cssert's main audience is server-rendered applications, where the CSS build @@ -74,25 +119,49 @@ attributes is reported as *dynamic-suspect* instead of verified. ``` cssert check [options] - --css Built CSS files (repeatable, comma-separated allowed) - --html HTML documents to check (repeatable) - --config Config file (default: cssert.config.{ts,js,mjs,json} in cwd) - --baseline Baseline file (default: .cssert/baseline.json when present) - --format human | json | github | sarif (default: human) - --output Write the report to a file instead of stdout - --attributes HTML attributes to scan (default: class) - --ignore Extra class or /regex/ to ignore (repeatable) - --allow Class or /regex/ that may be absent from the CSS (repeatable) - --strict-parse Treat CSS parse warnings as errors - --max-warnings Fail when parse warnings exceed - --fail-on-dynamic Also fail when dynamic-suspect tokens are found - --no-color Disable colours + --css Built CSS files (repeatable, comma-separated allowed) + --html HTML documents to check (repeatable) + --config Config file (default: cssert.config.{ts,js,mjs,json} in cwd) + --root Base directory for every relative path + --baseline Baseline file (default: .cssert/baseline.json when present) + --no-baseline Ignore the baseline for this run + --require-baseline Fail when the baseline file does not exist + --format human | json | github | sarif (default: human) + --annotate-occurrences github: annotate every usage, not one per class + --output Write the report to a file instead of stdout + --attributes HTML attributes to scan (default: class) + --ignore Extra class or /regex/ to ignore (repeatable) + --allow Class or /regex/ that may be absent from the CSS (repeatable) + --hook Class that is expected to have no styles (repeatable) + --strict-parse Treat CSS parse warnings as errors + --max-warnings Fail when parse warnings exceed + --fail-on-dynamic Also fail when dynamic-suspect tokens are found + --min-documents Fail when fewer than HTML documents were scanned + --min-stylesheets Fail when fewer than stylesheets were scanned + --no-color Disable colours ``` A class counts as defined when it appears in **any** selector of any stylesheet, including as an ancestor (`.group:hover .foo` defines `group`), so Tailwind's `group`/`peer` markers are not false positives. +### Where relative paths point + +Paths **in a config file** are resolved against the directory containing that +config file, the way eslint, vitest and tsc resolve theirs. Paths **passed as +flags** are resolved against the working directory. So the same config gives +the same result from anywhere, which matters when `node_modules` lives in a +subdirectory: + +```sh +cssert check # from the repo root +cd frontend && cssert check --config ../cssert.config.mjs # same result +``` + +Override the base with `root` in the config (relative to the config file) or +`--root` on the command line (relative to cwd; applies to flags too). Set +`resolveFrom: "cwd"` to get the pre-0.2 behaviour back. + ### `cssert baseline create | prune` The first run on an existing project usually reports many findings. Freeze @@ -104,9 +173,48 @@ npx @cssert/cli check ... # .cssert/baseline.json is applied automaticall npx @cssert/cli baseline prune # drop entries that have been fixed ``` -Commit `.cssert/baseline.json` and shrink it over time. Entries are keyed by -class name, so moving a usage to another template does not create a new -finding. +``` + --kind failing (default) | missing | dynamic-suspect | all + --dry-run Report what would change without writing the file +``` + +#### What to freeze, and what not to + +**A baseline is the list of debt you intend to pay off**, so only put things in +it that would otherwise fail the build. That is what `--kind failing` (the +default) does: `missing` findings, plus `dynamic-suspect` ones only when you +run with `--fail-on-dynamic`. + +Freezing dynamic-suspect findings otherwise makes the file rot. The key of such +an entry is the whole template expression: + +```json +{ "className": "{% if inv.status == 'active' %}badge-info{% else %}badge-ghost{% endif %}", + "kind": "dynamic-suspect" } +``` + +Edit one character of that condition and the key changes: the entry stops +matching and a stale line stays behind. The fix for those findings is to render +the page, not to freeze the expression. + +#### Recommended first run + +1. `cssert check` — look at the real list before freezing anything. +2. `cssert baseline create` — freeze what fails, and commit + `.cssert/baseline.json` in its own commit so the diff is reviewable. +3. Fix classes in batches. After each batch, `cssert baseline prune` drops the + entries that no longer occur and prints `N resolved, M remaining`; commit + that alongside the fix. `--dry-run` shows the same summary without writing. +4. When the file reaches zero entries, delete it and add `--require-baseline` + only if you want the absence itself to be an error. + +A baseline that does not exist yet is not an error: cssert notes it on stderr +and continues, so you can write the config before you freeze anything. Use +`--no-baseline` for a one-off run that shows everything, and +`--require-baseline` in CI if a missing file should fail. + +Entries are keyed by class name and kind, so moving a usage to another template +does not create a new finding. ### `cssert budget` @@ -133,9 +241,13 @@ export default defineConfig({ html: ["build/rendered/**/*.html", "templates/**/*.html"], ignore: [/^js-/, "legacy-banner"], allow: [], + hooks: ["sidebar-collapse-tooltip"], // queried from JS; correctly has no styles attributes: ["class", ":class"], baseline: ".cssert/baseline.json", + minDocuments: 27, // a truncated artefact must not pass budget: { snapshot: ".cssert/budget.json", maxDrop: "10%" }, + // root: "..", // rebase this file's relative paths + // resolveFrom: "cwd", // pre-0.2 resolution }); ``` @@ -143,6 +255,15 @@ Command-line flags override the config file. In JSON configs, strings of the form `"/^js-/"` are treated as regular expressions. TypeScript configs load natively on Node 22.18+/24+; on older runtimes install `jiti` or use `.mjs`. +`ignore`, `allow` and `hooks` all keep a class out of the report and differ in +what they record: + +| key | meaning | +| --- | --- | +| `ignore` | Not cssert's business (another library owns it, a script toggles it). Dropped before the comparison. | +| `allow` | Known to be absent, tolerated for now. | +| `hooks` | A script/test handle with no styling. Absence from the CSS is the *correct* state. | + Default ignore patterns (extend with `ignore`, disable with `useDefaultIgnore: false`): `js-*`, `is-*`, `has-*`, `active`, `disabled`, `wp-*`, `woocommerce-*`, `swiper-*`, `leaflet-*`. @@ -170,6 +291,21 @@ GitHub Actions annotations and SARIF for code scanning: sarif_file: cssert.sarif ``` +`--format github` emits **one annotation per class**, anchored at the first +occurrence, with the remaining count in the message +(`… is not defined in any stylesheet (and 65 other places).`). GitHub displays +only a limited number of annotations per check run, and the first run on an +existing project — when the report is longest — is exactly when one noisy class +would otherwise crowd out every other finding. Pass `--annotate-occurrences` +to annotate every usage instead. The human format always lists every +occurrence. + +When the HTML and the CSS are produced by different jobs, the checking job +receives both as artefacts — and an artefact that failed to download leaves an +empty directory. `--min-documents` makes that a failure instead of a green run; +see the [Django recipe](docs/recipes/django.md#5-github-actions) for a complete +three-job workflow. + ## Library usage ### Deriving findings @@ -181,7 +317,17 @@ const result = audit({ stylesheets: [{ path: "dist/app.css", css }], documents: [{ path: "index.html", html }], }); -result.findings; // [{ className, kind: "missing" | "dynamic-suspect", occurrences }] +result.findings; // [{ className, kind: "missing" | "dynamic-suspect", occurrences, sources }] +``` + +Label your documents with `source` and each finding reports the distinct labels +it was seen under. The CLI labels every document with the input glob that +matched it, so a finding whose `sources` contains only your template glob is a +page you never rendered: + +```sh +cssert check --html "build/rendered/**/*.html,templates/**/*.html" --format json \ + | jq '.findings[] | select(.sources == ["templates/**/*.html"]) | .className' ``` ### Explicit assertions @@ -239,6 +385,27 @@ comparable. - Linting, formatting or rewriting CSS - Verifying framework utility values +## Upgrading from 0.1 + +Four defaults changed. All four are about being usable rather than correct in +the abstract; each has an opt-out. + +| Change | Restore the old behaviour | +| --- | --- | +| Relative paths in a config file resolve against the config file's directory, not cwd | `resolveFrom: "cwd"` in the config | +| `--format github` emits one annotation per class, not per occurrence | `--annotate-occurrences` | +| A baseline file that does not exist is a note on stderr, not exit 2 | `--require-baseline` | +| `baseline create` freezes only the kinds that fail the check | `--kind all` | + +Check the first one before you upgrade: if your config file is not at the +directory you run cssert from, its globs now point somewhere else. Run +`cssert check` once and confirm the scanned document count is unchanged. + +Additions: `root` / `--root`, `--no-baseline`, `--min-documents`, +`--min-stylesheets`, `hooks` / `--hook`, `baseline --kind` / `--dry-run`, +`sources` on JSON findings, `documentsWithDynamic` in the summary, and +`--version` now naming the package. + ## Development ```sh diff --git a/docs/decisions.md b/docs/decisions.md index 75ebc6b..94a58b8 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -120,7 +120,9 @@ CI installs small. `--css`/`--html` patterns that match nothing exit with code 2 instead of reporting "no missing classes". A silently empty check is exactly the kind of -false reassurance principle 5 forbids. +false reassurance principle 5 forbids. (Extended in D30: a glob that matches +*fewer* files than expected is just as silent, which is what `--min-documents` +is for.) ## D18. Baselines are keyed by class name and kind, not by position @@ -131,10 +133,11 @@ one. Keying by file/line would make baselines churn on every unrelated edit. ## D19. An explicitly configured baseline must exist; the default may not -`check` silently skips the default `.cssert/baseline.json` when it is absent, -so first-time users are not forced to create one. A path given via -`--baseline` or the config file is a statement of intent, and a missing file -there is a usage error (exit 2) rather than a silent no-op. +**Superseded by D31 in 0.2.0.** `check` silently skips the default +`.cssert/baseline.json` when it is absent, so first-time users are not forced +to create one. A path given via `--baseline` or the config file is a statement +of intent, and a missing file there is a usage error (exit 2) rather than a +silent no-op. ## D20. Budget compares totals, only shrinkage fails, `--update` always passes @@ -156,11 +159,13 @@ several conditions, narrow with `under()` first. ## D22. CI formats emit one entry per occurrence -GitHub workflow commands and SARIF results are emitted per occurrence, not -per class, so every usage is annotated in the pull request. A finding without -occurrences (possible when a baseline or ignore list is edited by hand) is -still emitted once without a location. SARIF carries the class name as a -partial fingerprint so viewers can group alerts. +**Amended by D29 in 0.2.0: the GitHub format now folds to one annotation per +class; SARIF still emits one result per occurrence.** GitHub workflow commands +and SARIF results are emitted per occurrence, not per class, so every usage is +annotated in the pull request. A finding without occurrences (possible when a +baseline or ignore list is edited by hand) is still emitted once without a +location. SARIF carries the class name as a partial fingerprint so viewers can +group alerts. ## D23. Recipes render first, then check @@ -205,3 +210,81 @@ npm CLI (`npm install -g npm@latest`), which implements trusted publishing and provenance directly. The step is idempotent: it exits early when the version in `package.json` already exists on the registry, and `changeset tag` still creates the `@cssert/cli@x.y.z` git tag. + +## D28. Config paths resolve against the config file; flags against cwd + +Resolving every glob against the working directory made a config file's +meaning depend on where it was invoked from. In a repository whose npm root is +a subdirectory, `cd frontend && cssert check --config ../cssert.config.mjs` +matched nothing and said so in a way that reads like a build problem. eslint, +vitest and tsc all resolve config-relative, so that is now the default: +`Roots.config` for values read out of the config file, `Roots.flag` for values +typed on the command line, where "the file I pointed at from here" is the +obvious reading. Reported paths are relative to the base that found them, which +is what makes the output identical from any directory. + +`root` in a config file rebases only that file's own paths, because it is a +statement about the file. `--root` is given at invocation time and so rebases +flags too. `resolveFrom: "cwd"` restores the 0.1 behaviour for projects that +depended on it. + +## D29. The GitHub format annotates once per class + +A check run displays a limited number of annotations. One class used in 85 +places therefore hid every other finding — worst on the first run, when the +report is longest and the reader has the least context. The annotation is now +anchored at the first occurrence and carries the remaining count in its text +("and 65 other places"), so N classes produce N annotations. +`--annotate-occurrences` restores the old behaviour. The human format still +lists every occurrence: a terminal has no such cap, and the occurrence list is +how you find the class. + +SARIF is unchanged. Code scanning deduplicates by fingerprint and is built to +ingest large result sets, so per-occurrence results stay more useful there. + +## D30. cssert reports how much it could not verify + +`✓ no missing classes` means "no problems in what I was given", and the +distance between that and "no problems" is the whole risk of an SSR setup: a +page nobody rendered is silently out of scope. Three things make that visible +rather than assumed. The summary counts documents that still contain +unresolved class expressions, so a partly rendered input set says so on a green +run. `--min-documents`/`--min-stylesheets` turn a truncated input set into +`report.errors` and exit 1 — an artefact download that produced an empty +directory must not pass. And findings carry the input glob they were seen +under (`sources`), so a class that appears only under the template glob is a +page that was never rendered. + +Run-level errors live in `Report.errors` rather than being synthesised as +findings: they are not about a class, and a baseline must never be able to +suppress one. + +## D31. A missing baseline is a note, not an error + +D19 made a configured-but-absent baseline exit 2. That penalised the natural +order of work — write the config, then freeze the findings — and the first run +of every new project hit it. Absence now writes one line to stderr and the +check continues without suppression, which fails *louder* rather than quieter: +nothing is hidden. `--require-baseline` is there for pipelines that want the +absence itself to fail, and `--no-baseline` (or an empty path) is the explicit +"show me everything this time", replacing the workaround of renaming +`.cssert/` — which used to fail with EISDIR. + +## D32. `baseline create` freezes only what fails + +A baseline is the list of debt you intend to pay off. Freezing +dynamic-suspect findings by default put entries in it that could not fail the +build (without `--fail-on-dynamic`) and that rot: their key is the whole +template expression, so editing one character of a `{% if %}` condition changes +the key, the entry stops matching, and a stale line stays behind. The default +`--kind failing` freezes the kinds that fail under the current options; +`--kind all` keeps the old behaviour for projects that do run with +`--fail-on-dynamic`. + +## D33. `hooks` is `allow` with a documented reason + +A class that a script queries but never styles is not "unchecked" (`ignore`) +and not "tolerated for now" (`allow`) — its absence from the CSS is correct. +`hooks` behaves exactly like `allow` and exists so the config file says which +of the three a given entry is. Deliberately not a new finding kind: reporting +"this hook now has styles" would change the baseline schema for a rare case. diff --git a/docs/recipes/django.md b/docs/recipes/django.md index 2902729..31c07df 100644 --- a/docs/recipes/django.md +++ b/docs/recipes/django.md @@ -8,8 +8,81 @@ dynamic-suspect token. ## 1. Dump rendered HTML -Add a management command that renders URLs with the test client and writes -the responses to `build/rendered/`: +Two ways to do this. **Prefer the pytest one** if you already have tests: it +reuses your existing fixtures, so there is no demo dataset to keep in sync, and +it fails on a 500 before cssert ever runs. + +### 1a. From the test suite (recommended) + +Render the pages inside a test. The assertion on the status code is not a +bonus check — it is the thing that stops a broken page from reaching cssert as +a valid-looking 500 error page. + +```python +# tests/test_rendered_pages.py +from pathlib import Path + +import pytest +from django.test import Client + +OUT = Path("build/rendered") + +PAGES = [ + ("index.html", "/"), + ("pricing.html", "/pricing/"), + ("account/settings.html", "/account/settings/"), +] + + +@pytest.fixture +def demo_data(db, django_user_model): + """Reuse the fixtures the rest of the suite already builds.""" + user = django_user_model.objects.create_user("demo", password="x") + # ... create the objects the pages need, both branches of each condition + return user + + +@pytest.mark.parametrize("name,url", PAGES) +def test_page_renders_for_cssert(client: Client, demo_data, name, url): + client.force_login(demo_data) + response = client.get(url) + assert response.status_code == 200, f"{url} returned {response.status_code}" + + target = OUT / name + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(response.content) +``` + +```sh +pytest tests/test_rendered_pages.py # writes build/rendered/ +``` + +Keep `PAGES` in one place and count it: passing `--min-documents` with that +count turns "the render step silently produced nothing" into a failed check +rather than a green one. + +### Render both branches of every condition + +Rendering a page is not the same as covering it. `{% if %}` only emits the +branch your data happened to take, so the other branch's classes stay +unverified forever: + +```html + +``` + +With only unexpired invitations in the fixture, `badge-error` is never checked +— and it is exactly the kind of rarely rendered class a content-path mistake +drops. Build fixture data that exercises both sides of each condition that +affects a class: expired and current, checked and unchecked, permitted and +denied, empty list and populated list. Where a page has many such states, +render it several times under different names (`orders--empty.html`, +`orders--full.html`). + +### 1b. From a management command + +When there is no test suite to hang this off, add a command that renders URLs +with the test client and writes the responses to `build/rendered/`: ```python # myproject/core/management/commands/dump_rendered.py @@ -56,6 +129,10 @@ python manage.py loaddata fixtures/demo.json python manage.py dump_rendered ``` +This needs a demo dataset that CI can load and that someone has to maintain +alongside the test fixtures — which is why 1a is the better default when a test +suite exists. + Alternatively, render templates directly when a view needs no request state: ```python @@ -90,22 +167,107 @@ export default defineConfig({ css: ["static/dist/**/*.css"], html: ["build/rendered/**/*.html"], ignore: [/^js-/, /^admin-/, /^djdt-/], // Django admin / debug toolbar classes + minDocuments: 27, // as many pages as PAGES has entries +}); +``` + +These globs are relative to the config file, so put it at the repository root +next to `manage.py` and the command works from anywhere — including from a +`frontend/` subdirectory where `node_modules` lives: + +```json +// frontend/package.json +{ "scripts": { "cssert": "cssert check --config ../cssert.config.ts" } } +``` + +## 4. Render templates as a safety net + +Passing the raw templates *as well as* the rendered pages catches screens you +forgot to render. Their `{% %}` expressions come back as dynamic-suspect rather +than verified, but a plain `missing` finding that appears only in the template +glob is a page that was never rendered: + +```ts +export default defineConfig({ + css: ["static/dist/**/*.css"], + html: ["build/rendered/**/*.html", "templates/**/*.html"], + minDocuments: 27, }); ``` -## 4. GitHub Actions +Each finding in the JSON report carries the input glob it was seen under: + +```sh +cssert check --format json | jq '.findings[] + | select(.sources == ["templates/**/*.html"]) | .className' +``` + +Anything that lists prints a template whose rendered output is missing from +`build/rendered/`. + +## 5. GitHub Actions + +The CSS comes out of a Node job and the HTML out of a Python job with a +database, so the check is a third job that collects both artefacts: ```yaml -- run: python manage.py migrate && python manage.py loaddata fixtures/demo.json -- run: python manage.py dump_rendered -- run: npx @tailwindcss/cli -i static/src/app.css -o static/dist/app.css --minify -- run: npx @cssert/cli check --format github +jobs: + test: # renders the pages + runs-on: ubuntu-latest + services: + postgres: { image: postgres:17, env: { POSTGRES_PASSWORD: postgres } } + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-python@v6 + with: { python-version: "3.13" } + - run: pip install -r requirements-dev.txt + - run: pytest # writes build/rendered/ + - uses: actions/upload-artifact@v5 + with: { name: rendered-html, path: build/rendered, if-no-files-found: error } + + frontend-build: # builds the CSS + runs-on: ubuntu-latest + defaults: { run: { working-directory: frontend } } + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-node@v5 + with: { node-version: 22, cache: npm, cache-dependency-path: frontend/package-lock.json } + - run: npm ci + - run: npm run build + - uses: actions/upload-artifact@v5 + with: { name: built-css, path: frontend/dist/assets, if-no-files-found: error } + + cssert: + runs-on: ubuntu-latest + needs: [test, frontend-build] + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-node@v5 + with: { node-version: 22 } + - uses: actions/download-artifact@v5 + with: { name: rendered-html, path: build/rendered } + - uses: actions/download-artifact@v5 + with: { name: built-css, path: frontend/dist/assets } + - run: npx @cssert/cli check --format github ``` +Two details worth copying: + +- `if-no-files-found: error` on the uploads, and `minDocuments` / + `--min-documents` in the check. Between them, a render step or a download + that produced nothing fails the job instead of reporting a green run over + zero documents. +- The config file sits at the repository root and its globs + (`frontend/dist/assets/*.css`, `build/rendered/**/*.html`) are resolved + relative to it, so the same command works from the root, from `frontend/`, + and from an `npm run` script inside `frontend/package.json`. + ## Tips - Classes toggled from JavaScript (`is-open`, `active`) never appear in server - HTML and are ignored by default; add your own with `ignore`. + HTML and are ignored by default; add your own with `ignore`. For a class that + only exists as a `querySelector` handle and is *supposed* to have no styles, + use `hooks` instead — it keeps the reason in the config file. - If a Tailwind class shows up as missing, the fix is almost always in `@source`/`content`: the template directory (or an installed app's templates) is not being scanned. diff --git a/src/audit/derive.ts b/src/audit/derive.ts index 359437e..c587d09 100644 --- a/src/audit/derive.ts +++ b/src/audit/derive.ts @@ -21,7 +21,13 @@ export const DEFAULT_IGNORE: readonly RegExp[] = [ export interface AuditInput { stylesheets: { path: string; css: string }[]; - documents: { path: string; html: string }[]; + /** + * Documents to check. `source` labels where the document came from (the + * input glob that matched it, say `templates/**\/*.html`); findings carry + * the distinct labels of the documents they occur in, which separates + * "only appears in unrendered templates" from "appears in rendered output". + */ + documents: { path: string; html: string; source?: string }[]; /** Additional classes/patterns to skip. Appended to {@link DEFAULT_IGNORE}. */ ignore?: (string | RegExp)[]; /** Classes/patterns that are not reported even when absent from the CSS. */ @@ -46,6 +52,11 @@ export interface Finding { */ kind: "missing" | "dynamic-suspect"; occurrences: FindingOccurrence[]; + /** + * Distinct `source` labels of the documents this finding occurs in, sorted. + * Present only when the caller labelled its documents. + */ + sources?: string[]; } export interface AuditStats { @@ -55,6 +66,11 @@ export interface AuditStats { cssClasses: number; /** Distinct literal classes used across all documents (after ignore). */ htmlClasses: number; + /** + * Documents that still contain at least one unresolved class expression. + * A green run over these files proves less than over fully rendered ones. + */ + documentsWithDynamic: number; } export interface AuditResult { @@ -93,9 +109,10 @@ export function audit(input: AuditInput): AuditResult { warnings.push(...model.warnings); } - const missing = new Map(); - const dynamic = new Map(); + const missing = new Map(); + const dynamic = new Map(); const used = new Set(); + let documentsWithDynamic = 0; for (const doc of input.documents) { const extraction = extractClasses(doc.html, { @@ -105,24 +122,17 @@ export function audit(input: AuditInput): AuditResult { for (const [cls, occurrences] of extraction.classes) { used.add(cls); if (defined.has(cls) || matchesAny(cls, allow)) continue; - append(missing, cls, doc.path, occurrences); + append(missing, cls, doc, occurrences); } + if (extraction.dynamic.size > 0) documentsWithDynamic++; for (const [token, occurrences] of extraction.dynamic) { - append(dynamic, token, doc.path, occurrences); + append(dynamic, token, doc, occurrences); } } const findings: Finding[] = [ - ...[...missing].map(([className, occurrences]) => ({ - className, - kind: "missing" as const, - occurrences, - })), - ...[...dynamic].map(([className, occurrences]) => ({ - className, - kind: "dynamic-suspect" as const, - occurrences, - })), + ...[...missing].map(([className, acc]) => toFinding(className, "missing", acc)), + ...[...dynamic].map(([className, acc]) => toFinding(className, "dynamic-suspect", acc)), ]; return { @@ -133,21 +143,37 @@ export function audit(input: AuditInput): AuditResult { stylesheets: input.stylesheets.length, cssClasses: defined.size, htmlClasses: used.size, + documentsWithDynamic, }, }; } +interface Accumulator { + occurrences: FindingOccurrence[]; + sources: Set; +} + function append( - target: Map, + target: Map, key: string, - path: string, + doc: { path: string; source?: string }, occurrences: { line: number; column: number }[], ): void { - const list = target.get(key) ?? []; + const acc = target.get(key) ?? { occurrences: [], sources: new Set() }; + const { path } = doc; for (const o of occurrences) { - if (!list.some((e) => e.path === path && e.line === o.line && e.column === o.column)) { - list.push({ path, line: o.line, column: o.column }); + if ( + !acc.occurrences.some((e) => e.path === path && e.line === o.line && e.column === o.column) + ) { + acc.occurrences.push({ path, line: o.line, column: o.column }); } } - target.set(key, list); + if (doc.source !== undefined) acc.sources.add(doc.source); + target.set(key, acc); +} + +function toFinding(className: string, kind: Finding["kind"], acc: Accumulator): Finding { + const finding: Finding = { className, kind, occurrences: acc.occurrences }; + if (acc.sources.size > 0) finding.sources = [...acc.sources].sort(); + return finding; } diff --git a/src/cli/baseline.ts b/src/cli/baseline.ts index 7216a03..6ea55d9 100644 --- a/src/cli/baseline.ts +++ b/src/cli/baseline.ts @@ -1,5 +1,5 @@ import { existsSync, readFileSync } from "node:fs"; -import { resolve } from "node:path"; +import { isAbsolute, resolve } from "node:path"; import { type Baseline, createBaseline, @@ -7,7 +7,14 @@ import { pruneBaseline, serializeBaseline, } from "../audit/baseline.js"; -import { CHECK_USAGE, parseCheckArgs, resolveCheckOptions, runAudit } from "./check.js"; +import type { Finding } from "../audit/derive.js"; +import { + CHECK_USAGE, + type CheckOptions, + parseCheckArgs, + resolveCheckOptions, + runAudit, +} from "./check.js"; import { type CliContext, CliError, EXIT, type ExitCode } from "./context.js"; import { writeOutput } from "./files.js"; @@ -18,14 +25,60 @@ export const BASELINE_USAGE = `Usage: cssert baseline [options] create Freeze the current findings so that only new ones fail "cssert check" prune Remove entries whose finding no longer occurs -Options are the same as for "cssert check" (--css, --html, --config, ...). +Options are the same as for "cssert check" (--css, --html, --config, ...) plus: + --kind What to freeze: failing (default) | missing | + dynamic-suspect | all. "failing" means the kinds that + fail the check with the current options, so + dynamic-suspect is included only with --fail-on-dynamic. + --dry-run Report what would change without writing the file + The file is written to --baseline (default: ${DEFAULT_BASELINE_PATH}). ${CHECK_USAGE.slice(CHECK_USAGE.indexOf("Options:"))}`; +const BASELINE_OPTIONS = { + kind: { type: "string" }, + "dry-run": { type: "boolean" }, +} as const; + +/** Which finding kinds a `baseline create` run should freeze. */ +export type KindSelector = "failing" | "missing" | "dynamic-suspect" | "all"; + +export function parseKind(value: string | undefined): KindSelector { + if (value === undefined) return "failing"; + if (value === "failing" || value === "missing" || value === "dynamic-suspect" || value === "all") + return value; + throw new CliError( + `--kind must be one of failing, missing, dynamic-suspect, all (got ${value}).`, + ); +} + +/** + * Keep only the findings the selector asks for. + * + * The default freezes exactly what would fail the build. Freezing + * dynamic-suspect findings by default made baselines rot: their key is the + * whole template expression, so editing a condition changes the key and the + * entry silently stops matching. + */ +export function selectFindings( + findings: readonly Finding[], + selector: KindSelector, + failOnDynamic: boolean, +): Finding[] { + if (selector === "all") return [...findings]; + const kinds: Finding["kind"][] = + selector === "failing" + ? failOnDynamic + ? ["missing", "dynamic-suspect"] + : ["missing"] + : [selector]; + return findings.filter((f) => kinds.includes(f.kind)); +} + /** Read and validate a baseline file. Returns undefined when it does not exist. */ -export function readBaselineFile(path: string, cwd: string): Baseline | undefined { - const full = resolve(cwd, path); +export function readBaselineFile(path: string, base: string): Baseline | undefined { + const full = isAbsolute(path) ? path : resolve(base, path); if (!existsSync(full)) return undefined; let json: unknown; try { @@ -38,6 +91,11 @@ export function readBaselineFile(path: string, cwd: string): Baseline | undefine return parsed.baseline; } +/** The file `baseline create|prune` writes, ignoring `--no-baseline`. */ +function targetFor(options: CheckOptions): { path: string; base: string } { + return options.baseline ?? { path: DEFAULT_BASELINE_PATH, base: options.roots.config }; +} + export async function baselineCommand(argv: string[], ctx: CliContext): Promise { const [sub, ...rest] = argv; if (sub === undefined || sub === "-h" || sub === "--help") { @@ -47,30 +105,38 @@ export async function baselineCommand(argv: string[], ctx: CliContext): Promise< if (sub !== "create" && sub !== "prune") { throw new CliError(`Unknown baseline subcommand "${sub}".\n\n${BASELINE_USAGE}`); } - const parsed = parseCheckArgs(rest); + const parsed = parseCheckArgs(rest, BASELINE_OPTIONS, BASELINE_USAGE); if (parsed.help) { ctx.stdout.write(BASELINE_USAGE); return EXIT.ok; } + const kind = parseKind(parsed.raw.kind); + const dryRun = parsed.raw["dry-run"] === true; const options = await resolveCheckOptions(parsed.raw, ctx); - const path = options.baseline ?? DEFAULT_BASELINE_PATH; - const report = await runAudit(options, ctx); + const { path, base } = targetFor(options); + const report = await runAudit(options); if (sub === "create") { - const baseline = createBaseline(report.findings); - writeOutput(path, ctx.cwd, serializeBaseline(baseline)); + const selected = selectFindings(report.findings, kind, options.failOnDynamic); + const baseline = createBaseline(selected); + const skipped = report.findings.length - selected.length; + if (!dryRun) writeOutput(path, base, serializeBaseline(baseline)); ctx.stdout.write( - `Baseline written to ${path} (${baseline.entries.length} finding(s) frozen).\n`, + `${dryRun ? "Would write baseline to" : "Baseline written to"} ${path}` + + ` (${baseline.entries.length} finding(s) frozen` + + (skipped > 0 ? `, ${skipped} not failing the check left out` : "") + + `).\n`, ); return EXIT.ok; } - const existing = readBaselineFile(path, ctx.cwd); + const existing = readBaselineFile(path, base); if (!existing) throw new CliError(`Baseline not found: ${path}`); const { baseline, removed } = pruneBaseline(existing, report.findings); - writeOutput(path, ctx.cwd, serializeBaseline(baseline)); + if (!dryRun) writeOutput(path, base, serializeBaseline(baseline)); ctx.stdout.write( - `Baseline ${path} pruned: ${removed.length} resolved, ${baseline.entries.length} remaining.\n`, + `Baseline ${path}${dryRun ? " (dry run):" : " pruned:"}` + + ` ${removed.length} resolved, ${baseline.entries.length} remaining.\n`, ); for (const entry of removed) ctx.stdout.write(` - ${entry.className} (${entry.kind})\n`); return EXIT.ok; diff --git a/src/cli/budget.ts b/src/cli/budget.ts index 62e4d6e..f63f3e2 100644 --- a/src/cli/budget.ts +++ b/src/cli/budget.ts @@ -1,5 +1,5 @@ import { existsSync, readFileSync } from "node:fs"; -import { resolve } from "node:path"; +import { isAbsolute, resolve } from "node:path"; import { parseArgs } from "node:util"; import { type BudgetComparison, @@ -14,6 +14,7 @@ import { import { findConfigFile, loadConfigFile } from "./config.js"; import { type CliContext, CliError, EXIT, type ExitCode } from "./context.js"; import { loadFiles, writeOutput } from "./files.js"; +import { resolveRoots } from "./roots.js"; export const DEFAULT_SNAPSHOT_PATH = ".cssert/budget.json"; export const DEFAULT_MAX_DROP: MaxDrop = { percent: 10 }; @@ -26,12 +27,16 @@ the last snapshot. Catches a scan-path misconfiguration that empties the build. Options: --css Built CSS files (repeatable, comma-separated allowed) --snapshot Snapshot file (default: ${DEFAULT_SNAPSHOT_PATH}) + --root Base directory for every relative path --max-drop Allowed drop in classes/gzip size (default: 10%) --update Write the current measurement as the new snapshot --config Config file (default: cssert.config.{ts,js,mjs,json} in cwd) --format human | json (default: human) -h, --help Show this help +Paths written in a config file are relative to that file; paths passed as +flags are relative to the working directory. + When no snapshot exists one is created and the command exits 0. Exit codes: 0 within budget, 1 over budget, 2 usage/config error, 3 internal error. `; @@ -39,6 +44,7 @@ Exit codes: 0 within budget, 1 over budget, 2 usage/config error, 3 internal err interface BudgetArgs { css?: string[]; snapshot?: string; + root?: string; "max-drop"?: string; update?: boolean; config?: string; @@ -56,6 +62,7 @@ export async function budgetCommand(argv: string[], ctx: CliContext): Promise v.split(",")) .map((v) => v.trim()) .filter((v) => v !== ""); - const patterns = css.length > 0 ? css : (config.css ?? []); - const snapshotPath = values.snapshot ?? config.budget?.snapshot ?? DEFAULT_SNAPSHOT_PATH; + const cssSet = + css.length > 0 + ? { patterns: css, base: roots.flag } + : { patterns: config.css ?? [], base: roots.config }; + const snapshot = + values.snapshot !== undefined + ? { path: values.snapshot, base: roots.flag } + : config.budget?.snapshot !== undefined + ? { path: config.budget.snapshot, base: roots.config } + : { path: DEFAULT_SNAPSHOT_PATH, base: roots.config }; + const snapshotPath = snapshot.path; const maxDropText = values["max-drop"] ?? config.budget?.maxDrop; let maxDrop: MaxDrop = DEFAULT_MAX_DROP; if (maxDropText !== undefined) { @@ -91,13 +108,13 @@ export async function budgetCommand(argv: string[], ctx: CliContext): Promise ({ path: f.path, css: f.content }))); - const previous = readSnapshotFile(snapshotPath, ctx.cwd); + const previous = readSnapshotFile(snapshotPath, snapshot.base); const json = values.format === "json"; if (!previous) { - writeOutput(snapshotPath, ctx.cwd, serializeBudgetSnapshot(current)); + writeOutput(snapshotPath, snapshot.base, serializeBudgetSnapshot(current)); if (json) { ctx.stdout.write( `${JSON.stringify({ version: 1, tool: "cssert", created: true, snapshot: current }, null, 2)}\n`, @@ -112,7 +129,7 @@ export async function budgetCommand(argv: string[], ctx: CliContext): Promise Built CSS files (repeatable, comma-separated allowed) - --html HTML documents to check (repeatable) - --config Config file (default: cssert.config.{ts,js,mjs,json} in cwd) - --baseline Baseline file (default: .cssert/baseline.json when present) - --format human | json | github | sarif (default: human) - --output Write the report to a file instead of stdout - --attributes HTML attributes to scan (default: class) - --ignore Extra class or /regex/ to ignore (repeatable) - --allow Class or /regex/ that may be absent from the CSS (repeatable) - --strict-parse Treat CSS parse warnings as errors - --max-warnings Fail when parse warnings exceed - --fail-on-dynamic Also fail when dynamic-suspect tokens are found - --no-color Disable colours - -h, --help Show this help + --css Built CSS files (repeatable, comma-separated allowed) + --html HTML documents to check (repeatable) + --config Config file (default: cssert.config.{ts,js,mjs,json} in cwd) + --root Base directory for every relative path (default: the + config file's directory, or cwd without a config) + --baseline Baseline file (default: .cssert/baseline.json when present) + --no-baseline Ignore the baseline for this run + --require-baseline Fail when the baseline file does not exist + --format human | json | github | sarif (default: human) + --annotate-occurrences github: annotate every usage, not one per class + --output Write the report to a file instead of stdout + --attributes HTML attributes to scan (default: class) + --ignore Extra class or /regex/ to ignore (repeatable) + --allow Class or /regex/ that may be absent from the CSS (repeatable) + --hook Class that is expected to have no styles (repeatable) + --strict-parse Treat CSS parse warnings as errors + --max-warnings Fail when parse warnings exceed + --fail-on-dynamic Also fail when dynamic-suspect tokens are found + --min-documents Fail when fewer than HTML documents were scanned + --min-stylesheets Fail when fewer than stylesheets were scanned + --no-color Disable colours + -h, --help Show this help + +Paths written in a config file are relative to that file; paths passed as +flags are relative to the working directory. Exit codes: 0 no violations, 1 violations, 2 usage/config error, 3 internal error. `; +/** Where the baseline lives, or `undefined` when baselines are switched off. */ +export interface BaselineTarget { + path: string; + /** Directory the path is resolved against. */ + base: string; + /** Whether the user named this path (vs. the implicit default). */ + explicit: boolean; +} + export interface CheckOptions { - css: string[]; - html: string[]; + css: PatternSet; + html: PatternSet; + roots: Roots; config?: string; - baseline?: string; + baseline?: BaselineTarget; + requireBaseline: boolean; format: ReportFormat; + annotateOccurrences: boolean; output?: string; attributes?: string[]; ignore: (string | RegExp)[]; @@ -51,6 +75,8 @@ export interface CheckOptions { strictParse: boolean; maxWarnings?: number; failOnDynamic: boolean; + minDocuments?: number; + minStylesheets?: number; color: boolean; } @@ -62,8 +88,41 @@ function splitList(values: string[] | undefined): string[] { .filter((v) => v !== ""); } +type OptionConfig = NonNullable; + +const CHECK_OPTIONS: OptionConfig = { + css: { type: "string", multiple: true }, + html: { type: "string", multiple: true }, + config: { type: "string" }, + root: { type: "string" }, + baseline: { type: "string" }, + "no-baseline": { type: "boolean" }, + "require-baseline": { type: "boolean" }, + format: { type: "string" }, + "annotate-occurrences": { type: "boolean" }, + output: { type: "string" }, + attributes: { type: "string", multiple: true }, + ignore: { type: "string", multiple: true }, + allow: { type: "string", multiple: true }, + hook: { type: "string", multiple: true }, + "strict-parse": { type: "boolean" }, + "max-warnings": { type: "string" }, + "fail-on-dynamic": { type: "boolean" }, + "min-documents": { type: "string" }, + "min-stylesheets": { type: "string" }, + "no-color": { type: "boolean" }, + color: { type: "boolean" }, + help: { type: "boolean", short: "h" }, +}; + +/** + * Parse `check`-style arguments. `extra` adds command-specific options + * (`baseline` has `--kind` and `--dry-run`) without duplicating the list. + */ export function parseCheckArgs( argv: string[], + extra: OptionConfig = {}, + usage: string = CHECK_USAGE, ): { help: true } | { help: false; raw: RawCheckArgs } { let parsed: ReturnType; try { @@ -71,26 +130,10 @@ export function parseCheckArgs( args: argv, allowPositionals: false, strict: true, - options: { - css: { type: "string", multiple: true }, - html: { type: "string", multiple: true }, - config: { type: "string" }, - baseline: { type: "string" }, - format: { type: "string" }, - output: { type: "string" }, - attributes: { type: "string", multiple: true }, - ignore: { type: "string", multiple: true }, - allow: { type: "string", multiple: true }, - "strict-parse": { type: "boolean" }, - "max-warnings": { type: "string" }, - "fail-on-dynamic": { type: "boolean" }, - "no-color": { type: "boolean" }, - color: { type: "boolean" }, - help: { type: "boolean", short: "h" }, - }, + options: { ...CHECK_OPTIONS, ...extra }, }); } catch (error) { - throw new CliError(`${(error as Error).message}\n\n${CHECK_USAGE}`); + throw new CliError(`${(error as Error).message}\n\n${usage}`); } if (parsed.values.help) return { help: true }; return { help: false, raw: parsed.values as RawCheckArgs }; @@ -100,17 +143,35 @@ export interface RawCheckArgs { css?: string[]; html?: string[]; config?: string; + root?: string; baseline?: string; + "no-baseline"?: boolean; + "require-baseline"?: boolean; format?: string; + "annotate-occurrences"?: boolean; output?: string; attributes?: string[]; ignore?: string[]; allow?: string[]; + hook?: string[]; "strict-parse"?: boolean; "max-warnings"?: string; "fail-on-dynamic"?: boolean; + "min-documents"?: string; + "min-stylesheets"?: string; "no-color"?: boolean; color?: boolean; + kind?: string; + "dry-run"?: boolean; +} + +function integerFlag(value: string | undefined, flag: string): number | undefined { + if (value === undefined) return undefined; + const n = Number(value); + if (!Number.isInteger(n) || n < 0) { + throw new CliError(`${flag} must be a non-negative integer (got ${value}).`); + } + return n; } /** Merge config file values with command-line flags (flags win). */ @@ -120,90 +181,167 @@ export async function resolveCheckOptions( ): Promise { const configPath = raw.config ?? findConfigFile(ctx.cwd); const config: CssertConfig = configPath ? await loadConfigFile(configPath, ctx.cwd) : {}; + const roots = resolveRoots(ctx.cwd, configPath, config, raw.root); const cssFlags = splitList(raw.css); const htmlFlags = splitList(raw.html); const attributes = splitList(raw.attributes); - let maxWarnings = config.maxWarnings; - if (raw["max-warnings"] !== undefined) { - const n = Number(raw["max-warnings"]); - if (!Number.isInteger(n) || n < 0) { - throw new CliError( - `--max-warnings must be a non-negative integer (got ${raw["max-warnings"]}).`, - ); - } - maxWarnings = n; - } + const maxWarnings = integerFlag(raw["max-warnings"], "--max-warnings") ?? config.maxWarnings; + const minDocuments = integerFlag(raw["min-documents"], "--min-documents") ?? config.minDocuments; + const minStylesheets = + integerFlag(raw["min-stylesheets"], "--min-stylesheets") ?? config.minStylesheets; const colorFlag = raw["no-color"] ? false : raw.color === true ? true : undefined; const color = colorFlag ?? (ctx.env.NO_COLOR === undefined && ctx.env.FORCE_COLOR !== "0" && ctx.isTTY); const options: CheckOptions = { - css: cssFlags.length > 0 ? cssFlags : (config.css ?? []), - html: htmlFlags.length > 0 ? htmlFlags : (config.html ?? []), + css: + cssFlags.length > 0 + ? { patterns: cssFlags, base: roots.flag } + : { patterns: config.css ?? [], base: roots.config }, + html: + htmlFlags.length > 0 + ? { patterns: htmlFlags, base: roots.flag } + : { patterns: config.html ?? [], base: roots.config }, + roots, + requireBaseline: raw["require-baseline"] ?? config.requireBaseline ?? false, format: raw.format !== undefined ? parseFormat(raw.format, "--format") : (config.format ?? "human"), + annotateOccurrences: raw["annotate-occurrences"] ?? config.annotateOccurrences ?? false, ignore: [...(config.ignore ?? []), ...splitList(raw.ignore).map(toPattern)], - allow: [...(config.allow ?? []), ...splitList(raw.allow).map(toPattern)], + allow: [ + ...(config.allow ?? []), + ...(config.hooks ?? []), + ...splitList(raw.allow).map(toPattern), + ...splitList(raw.hook).map(toPattern), + ], useDefaultIgnore: config.useDefaultIgnore ?? true, strictParse: raw["strict-parse"] ?? config.strictParse ?? false, failOnDynamic: raw["fail-on-dynamic"] ?? config.failOnDynamic ?? false, color, }; if (configPath !== undefined) options.config = configPath; - const baseline = raw.baseline ?? config.baseline; + const baseline = resolveBaselineTarget(raw, config, roots); if (baseline !== undefined) options.baseline = baseline; + if (options.requireBaseline && baseline === undefined) { + throw new CliError("--require-baseline cannot be combined with --no-baseline."); + } if (raw.output !== undefined) options.output = raw.output; const attrs = attributes.length > 0 ? attributes : config.attributes; if (attrs !== undefined) options.attributes = attrs; if (maxWarnings !== undefined) options.maxWarnings = maxWarnings; + if (minDocuments !== undefined) options.minDocuments = minDocuments; + if (minStylesheets !== undefined) options.minStylesheets = minStylesheets; return options; } /** - * Apply the baseline file to a report. An explicitly configured baseline must - * exist; the default path is applied only when present. + * Decide which baseline file to use. `--no-baseline` (or an empty path) turns + * baselines off; a path from the config file is relative to the config, a path + * from the flag is relative to the working directory. + */ +function resolveBaselineTarget( + raw: RawCheckArgs, + config: CssertConfig, + roots: Roots, +): BaselineTarget | undefined { + if (raw["no-baseline"] === true) return undefined; + if (raw.baseline !== undefined) { + if (raw.baseline === "") return undefined; + return { path: raw.baseline, base: roots.flag, explicit: true }; + } + if (config.baseline !== undefined) { + if (config.baseline === "") return undefined; + return { path: config.baseline, base: roots.config, explicit: true }; + } + return { path: DEFAULT_BASELINE_PATH, base: roots.config, explicit: false }; +} + +/** + * Apply the baseline file to a report. A baseline that does not exist yet is + * not an error: writing the config before freezing the findings is the normal + * order of work. `--require-baseline` makes its absence fatal. */ export function applyBaselineFile(report: Report, options: CheckOptions, ctx: CliContext): Report { - const path = options.baseline ?? DEFAULT_BASELINE_PATH; - const baseline = readBaselineFile(path, ctx.cwd); + const target = options.baseline; + if (target === undefined) return report; + const baseline = readBaselineFile(target.path, target.base); if (!baseline) { - if (options.baseline !== undefined) - throw new CliError(`Baseline not found: ${options.baseline}`); + if (options.requireBaseline) throw new CliError(`Baseline not found: ${target.path}`); + if (target.explicit) { + ctx.stderr.write( + `cssert: no baseline at ${target.path}; ` + + 'run "cssert baseline create" to freeze the current findings.\n', + ); + } return report; } const { findings, suppressed } = applyBaseline(report.findings, baseline); - return { ...report, findings, baseline: { path, suppressed } }; + return { ...report, findings, baseline: { path: target.path, suppressed } }; } /** Run the audit described by `options` and return the report. */ -export async function runAudit(options: CheckOptions, ctx: CliContext): Promise { +export async function runAudit(options: CheckOptions): Promise { const [stylesheets, documents] = await Promise.all([ - loadFiles(options.css, ctx.cwd, "css"), - loadFiles(options.html, ctx.cwd, "html"), + loadFiles(options.css, "css"), + loadFiles(options.html, "html"), ]); const result = audit({ stylesheets: stylesheets.map((f) => ({ path: f.path, css: f.content })), - documents: documents.map((f) => ({ path: f.path, html: f.content })), + documents: documents.map((f) => ({ path: f.path, html: f.content, source: f.pattern })), ignore: options.ignore, allow: options.allow, useDefaultIgnore: options.useDefaultIgnore, ...(options.attributes ? { attributes: options.attributes } : {}), }); - return { findings: result.findings, warnings: result.warnings, stats: result.stats }; + const report: Report = { + findings: result.findings, + warnings: result.warnings, + stats: result.stats, + }; + const errors = coverageErrors(result.stats, options); + if (errors.length > 0) report.errors = errors; + return report; +} + +/** + * Check that enough input actually arrived. A CI job whose artefact download + * silently produced an empty directory would otherwise report a green run. + */ +export function coverageErrors( + stats: { documents: number; stylesheets: number }, + options: Pick, +): string[] { + const errors: string[] = []; + if (options.minDocuments !== undefined && stats.documents < options.minDocuments) { + errors.push( + `Only ${stats.documents} HTML document(s) were scanned; --min-documents requires at least ${options.minDocuments}.`, + ); + } + if (options.minStylesheets !== undefined && stats.stylesheets < options.minStylesheets) { + errors.push( + `Only ${stats.stylesheets} stylesheet(s) were scanned; --min-stylesheets requires at least ${options.minStylesheets}.`, + ); + } + return errors; } /** Render a report in the requested format. */ -export function renderReport(report: Report, format: ReportFormat, color: boolean): string { +export function renderReport( + report: Report, + format: ReportFormat, + color: boolean, + annotateOccurrences = false, +): string { switch (format) { case "human": return formatHuman(report, { color }); case "json": return formatJson(report); case "github": - return formatGithub(report); + return formatGithub(report, { annotateOccurrences }); case "sarif": return formatSarif(report, { version: readVersion() }); default: @@ -215,6 +353,7 @@ export function renderReport(report: Report, format: ReportFormat, color: boolea export function exitCodeFor(report: Report, options: CheckOptions): ExitCode { const hasMissing = report.findings.some((f) => f.kind === "missing"); const hasDynamic = report.findings.some((f) => f.kind === "dynamic-suspect"); + if ((report.errors ?? []).length > 0) return EXIT.violations; if (hasMissing) return EXIT.violations; if (hasDynamic && options.failOnDynamic) return EXIT.violations; if (options.strictParse && report.warnings.length > 0) return EXIT.violations; @@ -231,11 +370,16 @@ export async function checkCommand(argv: string[], ctx: CliContext): Promise { + const v = input[key]; + if (v === undefined) return undefined; + if (typeof v !== "string") throw new CliError(`Config ${display}: "${key}" must be a string.`); + return v; + }; + const integer = (key: string): number | undefined => { + const v = input[key]; + if (v === undefined) return undefined; + if (typeof v !== "number" || !Number.isInteger(v) || v < 0) { + throw new CliError(`Config ${display}: "${key}" must be a non-negative integer.`); + } + return v; + }; + const css = list("css"); if (css) config.css = css; const html = list("html"); @@ -114,19 +129,36 @@ function normalizeConfig(raw: unknown, display: string): CssertConfig { if (ignore) config.ignore = ignore; const allow = patterns("allow"); if (allow) config.allow = allow; + const hooks = patterns("hooks"); + if (hooks) config.hooks = hooks; const useDefaultIgnore = bool("useDefaultIgnore"); if (useDefaultIgnore !== undefined) config.useDefaultIgnore = useDefaultIgnore; const strictParse = bool("strictParse"); if (strictParse !== undefined) config.strictParse = strictParse; const failOnDynamic = bool("failOnDynamic"); if (failOnDynamic !== undefined) config.failOnDynamic = failOnDynamic; + const requireBaseline = bool("requireBaseline"); + if (requireBaseline !== undefined) config.requireBaseline = requireBaseline; + const annotateOccurrences = bool("annotateOccurrences"); + if (annotateOccurrences !== undefined) config.annotateOccurrences = annotateOccurrences; - if (input.baseline !== undefined) { - if (typeof input.baseline !== "string") { - throw new CliError(`Config ${display}: "baseline" must be a string.`); + const root = str("root"); + if (root !== undefined) config.root = root; + if (input.resolveFrom !== undefined) { + if (input.resolveFrom !== "config" && input.resolveFrom !== "cwd") { + throw new CliError( + `Config ${display}: "resolveFrom" must be "config" or "cwd" (got ${String(input.resolveFrom)}).`, + ); } - config.baseline = input.baseline; + config.resolveFrom = input.resolveFrom; } + const minDocuments = integer("minDocuments"); + if (minDocuments !== undefined) config.minDocuments = minDocuments; + const minStylesheets = integer("minStylesheets"); + if (minStylesheets !== undefined) config.minStylesheets = minStylesheets; + + const baseline = str("baseline"); + if (baseline !== undefined) config.baseline = baseline; if (input.format !== undefined) { config.format = parseFormat(input.format, display); } diff --git a/src/cli/files.ts b/src/cli/files.ts index c453411..9c53902 100644 --- a/src/cli/files.ts +++ b/src/cli/files.ts @@ -1,44 +1,61 @@ import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; -import { dirname, relative, resolve } from "node:path"; +import { dirname, isAbsolute, relative, resolve } from "node:path"; import { glob } from "tinyglobby"; import { CliError } from "./context.js"; +import type { PatternSet } from "./roots.js"; export interface LoadedFile { - /** Path relative to cwd, with forward slashes, for reporting. */ + /** Path relative to the base directory, with forward slashes, for reporting. */ path: string; + /** The input glob that matched this file, for provenance in reports. */ + pattern: string; content: string; } /** - * Expand globs relative to `cwd` and read every match. Throws a usage error - * when nothing matches so a misconfigured path fails loudly instead of + * Expand globs relative to `set.base` and read every match. Throws a usage + * error when nothing matches so a misconfigured path fails loudly instead of * producing an empty (and therefore always passing) check. + * + * Each positive pattern is expanded separately (with the negative patterns of + * the set applied to all of them) so every file can name the glob it came + * from; the resulting set is the same as a single combined expansion. */ -export async function loadFiles( - patterns: string[], - cwd: string, - label: string, -): Promise { - if (patterns.length === 0) { +export async function loadFiles(set: PatternSet, label: string): Promise { + if (set.patterns.length === 0) { throw new CliError(`No ${label} files specified. Use --${label} or a config file.`); } - const matches = await glob(patterns, { cwd, onlyFiles: true, expandDirectories: false }); - if (matches.length === 0) { - throw new CliError(`No ${label} files matched: ${patterns.join(", ")}`); + const negative = set.patterns.filter((p) => p.startsWith("!")); + const positive = set.patterns.filter((p) => !p.startsWith("!")); + + const attribution = new Map(); + for (const pattern of positive) { + const matches = await glob([pattern, ...negative], { + cwd: set.base, + onlyFiles: true, + expandDirectories: false, + }); + for (const match of matches) { + if (!attribution.has(match)) attribution.set(match, pattern); + } + } + if (attribution.size === 0) { + throw new CliError(`No ${label} files matched: ${set.patterns.join(", ")}`); } - matches.sort(); - return matches.map((match) => { - const full = resolve(cwd, match); + + return [...attribution.keys()].sort().map((match) => { + const full = resolve(set.base, match); return { - path: toPosix(relative(cwd, full)) || match, + path: toPosix(relative(set.base, full)) || match, + pattern: attribution.get(match) as string, content: readFileSync(full, "utf8"), }; }); } -/** Write text to `path` (relative to cwd), creating parent directories. */ -export function writeOutput(path: string, cwd: string, text: string): void { - const full = resolve(cwd, path); +/** Write text to `path` (relative to `base`), creating parent directories. */ +export function writeOutput(path: string, base: string, text: string): void { + const full = isAbsolute(path) ? path : resolve(base, path); mkdirSync(dirname(full), { recursive: true }); writeFileSync(full, text, "utf8"); } diff --git a/src/cli/main.ts b/src/cli/main.ts index a2be8a4..b9ef5c9 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -2,7 +2,7 @@ import { baselineCommand } from "./baseline.js"; import { budgetCommand } from "./budget.js"; import { checkCommand } from "./check.js"; import { type CliContext, CliError, defaultContext, EXIT, type ExitCode } from "./context.js"; -import { readVersion } from "./version.js"; +import { PACKAGE_NAME, versionLine } from "./version.js"; export const USAGE = `Usage: cssert [options] @@ -12,6 +12,8 @@ Commands: budget Guard against a stylesheet that shrank unexpectedly Run "cssert --help" for command options. + +cssert is the binary of the ${PACKAGE_NAME} package. `; type Command = (argv: string[], ctx: CliContext) => Promise; @@ -36,7 +38,7 @@ export async function runCli( return EXIT.ok; } if (command === "-v" || command === "--version") { - ctx.stdout.write(`${readVersion()}\n`); + ctx.stdout.write(`${versionLine()}\n`); return EXIT.ok; } const handler = COMMANDS[command]; diff --git a/src/cli/roots.ts b/src/cli/roots.ts new file mode 100644 index 0000000..95ad6ed --- /dev/null +++ b/src/cli/roots.ts @@ -0,0 +1,56 @@ +import { dirname, resolve } from "node:path"; +import type { CssertConfig } from "../config.js"; + +/** + * The directories that relative paths are resolved against. + * + * Paths written in a config file are relative to that file (`config`), the way + * eslint, vitest and tsc treat their own configs. Paths typed on the command + * line stay relative to the working directory (`flag`), which is what "I + * pointed at a file here" means. `root` in the config file rebases the config + * side only; `--root`, given at invocation time, collapses both onto the same + * directory. + */ +export interface Roots { + /** Base for values that came from the config file. */ + config: string; + /** Base for values that came from command-line flags. */ + flag: string; +} + +/** Where a path or glob came from, which decides its base directory. */ +export type Origin = keyof Roots; + +/** A set of globs together with the directory they are resolved against. */ +export interface PatternSet { + patterns: string[]; + base: string; +} + +/** + * Work out the base directories for one CLI run. + * + * @param cwd Working directory of the process. + * @param configPath Absolute path of the loaded config file, if any. + * @param config The loaded config (for `root` / `resolveFrom`). + * @param rootFlag Value of `--root`, relative to `cwd`. + */ +export function resolveRoots( + cwd: string, + configPath: string | undefined, + config: Pick, + rootFlag: string | undefined, +): Roots { + if (rootFlag !== undefined && rootFlag !== "") { + const root = resolve(cwd, rootFlag); + return { config: root, flag: root }; + } + const configDir = configPath === undefined ? undefined : dirname(resolve(cwd, configPath)); + if (config.root !== undefined && config.root !== "") { + // `root` in a config file rebases that file's own paths; it does not move + // command-line globs, which keep meaning "relative to where I am". + return { config: resolve(configDir ?? cwd, config.root), flag: cwd }; + } + const useConfigDir = configDir !== undefined && (config.resolveFrom ?? "config") === "config"; + return { config: useConfigDir ? configDir : cwd, flag: cwd }; +} diff --git a/src/cli/version.ts b/src/cli/version.ts index 0cfa048..2be60f0 100644 --- a/src/cli/version.ts +++ b/src/cli/version.ts @@ -1,5 +1,8 @@ import { readFileSync } from "node:fs"; +/** The npm package this binary was installed from. The binary is `cssert`. */ +export const PACKAGE_NAME = "@cssert/cli"; + /** Read the package version from the nearest package.json (dist/cli → root). */ export function readVersion(): string { for (const rel of ["../../package.json", "../package.json"]) { @@ -8,10 +11,19 @@ export function readVersion(): string { name?: string; version?: string; }; - if (pkg.name === "@cssert/cli" && typeof pkg.version === "string") return pkg.version; + if (pkg.name === PACKAGE_NAME && typeof pkg.version === "string") return pkg.version; } catch { // try the next candidate } } return "unknown"; } + +/** + * The `--version` line. It names the package as well as the binary: the two + * differ (npm rejects the bare name `cssert`), and "is this really the thing I + * installed?" should be answerable without opening the README. + */ +export function versionLine(): string { + return `cssert ${readVersion()} (${PACKAGE_NAME})`; +} diff --git a/src/config.ts b/src/config.ts index 4a8e365..81e2133 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1,12 +1,28 @@ /** * Configuration accepted by `cssert.config.{ts,js,mjs,json}` and by the CLI. * Every field is optional; command-line flags take precedence. + * + * Relative paths in a config file are resolved against the directory that + * contains the config file, not against the working directory. Set + * {@link CssertConfig.root} to override that base, or + * `resolveFrom: "cwd"` to restore the pre-0.2 behaviour. */ export interface CssertConfig { /** Globs for built CSS files. */ css?: string[]; /** Globs for HTML documents to check. */ html?: string[]; + /** + * Base directory for the relative paths in this file. Relative to the + * config file's own directory. Default: the config file's directory. + */ + root?: string; + /** + * Where relative paths in this file are resolved from when `root` is unset. + * `"config"` (default) uses the config file's directory; `"cwd"` restores + * the 0.1 behaviour of resolving against the working directory. + */ + resolveFrom?: "config" | "cwd"; /** * Classes to skip entirely, in addition to the default ignore list. * Strings match exactly; a string of the form `/pattern/flags` is treated as @@ -15,20 +31,38 @@ export interface CssertConfig { ignore?: (string | RegExp)[]; /** Classes that are not reported as missing even when absent from the CSS. */ allow?: (string | RegExp)[]; + /** + * Classes that exist only as script/test hooks and are *expected* to have no + * styles. Behaves like {@link CssertConfig.allow}; the separate key records + * the intent ("correct that the CSS does not define this") rather than + * "not checked". + */ + hooks?: (string | RegExp)[]; /** Set to false to disable the built-in ignore list. */ useDefaultIgnore?: boolean; /** HTML attributes to scan. Default `["class"]`. */ attributes?: string[]; /** Baseline file. Default `.cssert/baseline.json`, applied when present. */ baseline?: string; + /** Fail when the baseline file is absent instead of continuing without one. */ + requireBaseline?: boolean; /** Report format. Default `human`. */ format?: "human" | "json" | "github" | "sarif"; + /** + * Emit one `--format github` annotation per occurrence instead of one per + * class. Default false; GitHub caps the annotations it displays per run. + */ + annotateOccurrences?: boolean; /** Treat parse warnings as errors. */ strictParse?: boolean; /** Fail when the number of parse warnings exceeds this value. */ maxWarnings?: number; /** Also fail (exit 1) when dynamic-suspect tokens are found. Default false. */ failOnDynamic?: boolean; + /** Fail when fewer than this many HTML documents were scanned. */ + minDocuments?: number; + /** Fail when fewer than this many stylesheets were scanned. */ + minStylesheets?: number; /** Settings for `cssert budget`. */ budget?: { /** Snapshot file. Default `.cssert/budget.json`. */ diff --git a/src/report/github.ts b/src/report/github.ts index a7b56ac..b556e9c 100644 --- a/src/report/github.ts +++ b/src/report/github.ts @@ -1,5 +1,5 @@ import type { Warning } from "../core/types.js"; -import { countFindings, type Report } from "./types.js"; +import { countFindings, type Report, type ReportOptions } from "./types.js"; /** * Escape text for a GitHub Actions workflow command message. @@ -24,28 +24,50 @@ function command( return `::${level}${parts.length > 0 ? ` ${parts.join(",")}` : ""}::${escapeData(message)}`; } +/** The annotation body for a finding, with the count of further occurrences. */ +function messageFor(finding: Report["findings"][number], others: number): string { + const elsewhere = others > 0 ? ` (and ${others} other place${others === 1 ? "" : "s"})` : ""; + return finding.kind === "missing" + ? `Class "${finding.className}" is used here but is not defined in any stylesheet${elsewhere}.` + : `Token "${finding.className}" contains template syntax and cannot be checked${elsewhere}; the class name is probably built dynamically.`; +} + /** * Render the report as GitHub Actions workflow commands so findings show up * as inline annotations on the pull request, followed by a plain summary line. + * + * One annotation per class by default, anchored at its first occurrence: a + * check run displays only a limited number of annotations, and a single class + * used in 85 places would otherwise crowd out every other finding. Pass + * `annotateOccurrences` to annotate every usage instead. */ -export function formatGithub(report: Report): string { +export function formatGithub(report: Report, opts: ReportOptions = {}): string { const lines: string[] = []; for (const finding of report.findings) { const isMissing = finding.kind === "missing"; const level = isMissing ? "error" : "warning"; const title = isMissing ? "cssert: missing class" : "cssert: dynamic class"; - const message = isMissing - ? `Class "${finding.className}" is used here but is not defined in any stylesheet.` - : `Token "${finding.className}" contains template syntax and cannot be checked; the class name is probably built dynamically.`; if (finding.occurrences.length === 0) { - lines.push(command(level, { title }, message)); + lines.push(command(level, { title }, messageFor(finding, 0))); + continue; } - for (const occ of finding.occurrences) { + const annotated = opts.annotateOccurrences + ? finding.occurrences + : finding.occurrences.slice(0, 1); + const others = opts.annotateOccurrences ? 0 : finding.occurrences.length - 1; + for (const occ of annotated) { lines.push( - command(level, { file: occ.path, line: occ.line, col: occ.column, title }, message), + command( + level, + { file: occ.path, line: occ.line, col: occ.column, title }, + messageFor(finding, others), + ), ); } } + for (const error of report.errors ?? []) { + lines.push(command("error", { title: "cssert: input coverage" }, error)); + } for (const warning of report.warnings) { lines.push(command("warning", warningProps(warning), warning.message)); } @@ -53,6 +75,9 @@ export function formatGithub(report: Report): string { lines.push( `cssert: ${counts.missing} missing, ${counts.dynamicSuspect} dynamic-suspect, ${report.warnings.length} warning(s)` + ` · scanned ${report.stats.documents} document(s) / ${report.stats.stylesheets} stylesheet(s)` + + (report.stats.documentsWithDynamic > 0 + ? ` · ${report.stats.documentsWithDynamic} document(s) still contain unresolved class expressions` + : "") + (report.baseline && report.baseline.suppressed > 0 ? ` · ${report.baseline.suppressed} suppressed by baseline` : ""), diff --git a/src/report/human.ts b/src/report/human.ts index ef3457b..167f717 100644 --- a/src/report/human.ts +++ b/src/report/human.ts @@ -50,6 +50,10 @@ export function formatHuman(report: Report, opts: ReportOptions = {}): string { } } + for (const error of report.errors ?? []) { + lines.push(` ${paint("✗", ANSI.red)} ${error}`); + } + if (report.warnings.length > 0) { if (lines.length > 0) lines.push(""); lines.push(` ${paint(`${report.warnings.length} parse warning(s):`, ANSI.yellow)}`); @@ -65,12 +69,25 @@ export function formatHuman(report: Report, opts: ReportOptions = {}): string { report.baseline && report.baseline.suppressed > 0 ? ` · ${report.baseline.suppressed} suppressed by baseline` : ""; - if (report.findings.length === 0) { - lines.push(` ${paint("✓", ANSI.green)} no missing classes · ${scanned}${baseline}`); + // What the run could not verify is part of the result, not a footnote: + // a document with unresolved class expressions was only partly checked. + const unresolved = + report.stats.documentsWithDynamic > 0 + ? ` · ${paint( + `${report.stats.documentsWithDynamic} document(s) still contain unresolved class expressions`, + ANSI.yellow, + )}` + : ""; + if (report.findings.length === 0 && (report.errors ?? []).length === 0) { + lines.push( + ` ${paint("✓", ANSI.green)} no missing classes · ${scanned}${unresolved}${baseline}`, + ); + } else if (report.findings.length === 0) { + lines.push(` ${paint("input coverage check failed", ANSI.red)} · ${scanned}${baseline}`); } else { const summary = `${counts.missing} missing, ${counts.dynamicSuspect} dynamic-suspect`; lines.push( - ` ${paint(summary, counts.missing > 0 ? ANSI.red : ANSI.yellow)} · ${scanned}${baseline}`, + ` ${paint(summary, counts.missing > 0 ? ANSI.red : ANSI.yellow)} · ${scanned}${unresolved}${baseline}`, ); } return `${lines.join("\n")}\n`; diff --git a/src/report/json.ts b/src/report/json.ts index 8e48587..8d8438f 100644 --- a/src/report/json.ts +++ b/src/report/json.ts @@ -10,10 +10,14 @@ export interface JsonReport { warnings: number; documents: number; stylesheets: number; + /** Documents that still contain unresolved class expressions. */ + documentsWithDynamic: number; suppressedByBaseline: number; }; findings: Report["findings"]; warnings: Report["warnings"]; + /** Run-level failures not tied to a class (input coverage). */ + errors: string[]; } /** Build the JSON report object. */ @@ -28,10 +32,12 @@ export function toJsonReport(report: Report): JsonReport { warnings: report.warnings.length, documents: report.stats.documents, stylesheets: report.stats.stylesheets, + documentsWithDynamic: report.stats.documentsWithDynamic, suppressedByBaseline: report.baseline?.suppressed ?? 0, }, findings: report.findings, warnings: report.warnings, + errors: report.errors ?? [], }; } diff --git a/src/report/sarif.ts b/src/report/sarif.ts index 72de4c3..cb37f79 100644 --- a/src/report/sarif.ts +++ b/src/report/sarif.ts @@ -4,6 +4,7 @@ export const SARIF_RULES = { missing: "cssert/missing-class", dynamic: "cssert/dynamic-class", parse: "cssert/parse-warning", + coverage: "cssert/input-coverage", } as const; interface SarifLocation { @@ -80,6 +81,13 @@ export function toSarif(report: Report, opts: ReportOptions = {}): SarifLog { results.push({ ...base, locations: [location(occ.path, occ.line, occ.column)] }); } } + for (const error of report.errors ?? []) { + results.push({ + ruleId: SARIF_RULES.coverage, + level: "error", + message: { text: error }, + }); + } for (const w of report.warnings) { const result: SarifResult = { ruleId: SARIF_RULES.parse, @@ -116,6 +124,16 @@ export function toSarif(report: Report, opts: ReportOptions = {}): SarifLog { helpUri: `${HOMEPAGE}#readme`, defaultConfiguration: { level: "warning" }, }, + { + id: SARIF_RULES.coverage, + name: "InputCoverage", + shortDescription: { text: "Fewer inputs were checked than required" }, + fullDescription: { + text: "The run scanned fewer documents or stylesheets than --min-documents/--min-stylesheets requires. A check over an empty or truncated input set passes without proving anything.", + }, + helpUri: `${HOMEPAGE}#readme`, + defaultConfiguration: { level: "error" }, + }, { id: SARIF_RULES.parse, name: "ParseWarning", diff --git a/src/report/types.ts b/src/report/types.ts index 3a7e201..6e49da4 100644 --- a/src/report/types.ts +++ b/src/report/types.ts @@ -8,6 +8,12 @@ export interface Report { stats: AuditStats; /** Present when a baseline file was applied. */ baseline?: { path: string; suppressed: number }; + /** + * Run-level failures that are not tied to a class, such as "fewer documents + * were scanned than `--min-documents` requires". Reported by every format + * and counted towards the exit code. + */ + errors?: string[]; } export type ReportFormat = "human" | "json" | "github" | "sarif"; @@ -17,6 +23,12 @@ export interface ReportOptions { color?: boolean; /** Tool version to embed (SARIF). */ version?: string; + /** + * Emit one annotation per occurrence instead of one per class (github). + * Default false: GitHub caps how many annotations it displays per run, and + * one noisy class would otherwise hide every other finding. + */ + annotateOccurrences?: boolean; } /** Count findings by kind. */ diff --git a/test/audit/derive.test.ts b/test/audit/derive.test.ts index 2614cf4..038ffcf 100644 --- a/test/audit/derive.test.ts +++ b/test/audit/derive.test.ts @@ -40,7 +40,13 @@ describe("audit", () => { occurrences: [{ path: "two.html", line: 1, column: 24 }], }, ]); - expect(result.stats).toEqual({ documents: 2, stylesheets: 2, cssClasses: 4, htmlClasses: 5 }); + expect(result.stats).toEqual({ + documents: 2, + stylesheets: 2, + cssClasses: 4, + htmlClasses: 5, + documentsWithDynamic: 1, + }); expect(result.warnings).toEqual([]); }); diff --git a/test/cli/baseline-budget.test.ts b/test/cli/baseline-budget.test.ts index 9d38d1e..981f2c5 100644 --- a/test/cli/baseline-budget.test.ts +++ b/test/cli/baseline-budget.test.ts @@ -21,23 +21,22 @@ describe("cssert baseline", () => { expect((await sb.run(["baseline", "wat"])).code).toBe(2); }); - it("creates a baseline that check applies automatically", async () => { + it("freezes only the findings that fail the check, and check applies them", async () => { expect((await sb.run(["check", ...base])).code).toBe(1); const create = await sb.run(["baseline", "create", ...base]); expect(create.code).toBe(0); expect(create.stdout).toContain(".cssert/baseline.json"); - expect(create.stdout).toContain("3 finding(s) frozen"); + expect(create.stdout).toContain("2 finding(s) frozen, 1 not failing the check left out"); const baseline = JSON.parse(sb.read(".cssert/baseline.json")); expect(baseline.entries).toEqual([ { className: "old-1", kind: "missing" }, { className: "old-2", kind: "missing" }, - { className: "text-{{ color }}", kind: "dynamic-suspect" }, ]); const check = await sb.run(["check", ...base]); expect(check.code).toBe(0); - expect(check.stdout).toContain("3 suppressed by baseline"); + expect(check.stdout).toContain("2 suppressed by baseline"); // A new missing class still fails. sb.write("templates/new.html", `

`); @@ -47,12 +46,67 @@ describe("cssert baseline", () => { expect(again.stdout).not.toContain("old-1"); }); - it("writes to a custom --baseline path and requires it to exist for check", async () => { + it("freezes dynamic-suspect findings with --fail-on-dynamic or --kind all", async () => { + const dynamic = { className: "text-{{ color }}", kind: "dynamic-suspect" }; + + await sb.run(["baseline", "create", ...base, "--fail-on-dynamic"]); + expect(JSON.parse(sb.read(".cssert/baseline.json")).entries).toContainEqual(dynamic); + + await sb.run(["baseline", "create", ...base, "--kind", "all"]); + expect(JSON.parse(sb.read(".cssert/baseline.json")).entries).toContainEqual(dynamic); + + await sb.run(["baseline", "create", ...base, "--kind", "dynamic-suspect"]); + expect(JSON.parse(sb.read(".cssert/baseline.json")).entries).toEqual([dynamic]); + + expect((await sb.run(["baseline", "create", ...base, "--kind", "wat"])).code).toBe(2); + }); + + it("previews without writing with --dry-run", async () => { + const create = await sb.run(["baseline", "create", ...base, "--dry-run"]); + expect(create.code).toBe(0); + expect(create.stdout).toContain("Would write baseline to .cssert/baseline.json"); + expect(existsSync(join(sb.dir, ".cssert/baseline.json"))).toBe(false); + + await sb.run(["baseline", "create", ...base]); + sb.write("templates/bad.html", `

`); + const prune = await sb.run(["baseline", "prune", ...base, "--dry-run"]); + expect(prune.stdout).toContain("(dry run): 1 resolved, 1 remaining"); + expect(JSON.parse(sb.read(".cssert/baseline.json")).entries).toHaveLength(2); + }); + + it("writes to a custom --baseline path, tolerating its absence until created", async () => { const custom = ["--baseline", "ci/known.json"]; - expect((await sb.run(["check", ...base, ...custom])).code).toBe(2); + const before = await sb.run(["check", ...base, ...custom]); + expect(before.code).toBe(1); // the findings, not a usage error + expect(before.stderr).toContain("no baseline at ci/known.json"); + expect((await sb.run(["check", ...base, ...custom, "--require-baseline"])).code).toBe(2); + expect((await sb.run(["baseline", "create", ...base, ...custom])).code).toBe(0); expect(existsSync(join(sb.dir, "ci/known.json"))).toBe(true); expect((await sb.run(["check", ...base, ...custom])).code).toBe(0); + expect((await sb.run(["check", ...base, ...custom, "--require-baseline"])).code).toBe(0); + }); + + it("ignores the baseline with --no-baseline and an empty --baseline", async () => { + await sb.run(["baseline", "create", ...base]); + expect((await sb.run(["check", ...base])).code).toBe(0); + expect((await sb.run(["check", ...base, "--no-baseline"])).code).toBe(1); + expect((await sb.run(["check", ...base, "--baseline", ""])).code).toBe(1); + expect((await sb.run(["check", ...base, "--no-baseline", "--require-baseline"])).code).toBe(2); + }); + + it("continues when a config-declared baseline does not exist yet", async () => { + sb.write( + "cssert.config.json", + JSON.stringify({ + css: ["dist/app.css"], + html: ["templates/*.html"], + baseline: ".cssert/baseline.json", + }), + ); + const res = await sb.run(["check"]); + expect(res.code).toBe(1); + expect(res.stderr).toContain('run "cssert baseline create"'); }); it("reads the baseline path from the config file", async () => { @@ -74,7 +128,7 @@ describe("cssert baseline", () => { sb.write("templates/bad.html", `
`); const prune = await sb.run(["baseline", "prune", ...base]); expect(prune.code).toBe(0); - expect(prune.stdout).toContain("2 resolved, 1 remaining"); + expect(prune.stdout).toContain("1 resolved, 1 remaining"); expect(prune.stdout).toContain("- old-1 (missing)"); expect(JSON.parse(sb.read(".cssert/baseline.json")).entries).toEqual([ { className: "old-2", kind: "missing" }, diff --git a/test/cli/check.test.ts b/test/cli/check.test.ts index 62bedae..1d514b4 100644 --- a/test/cli/check.test.ts +++ b/test/cli/check.test.ts @@ -19,10 +19,11 @@ describe("cssert (top level)", () => { expect(help.stdout).toContain("Usage: cssert "); }); - it("prints the version", async () => { + it("prints the version with the package name", async () => { const res = await sb.run(["--version"]); expect(res.code).toBe(0); - expect(res.stdout.trim()).toMatch(/^\d+\.\d+\.\d+/); + expect(res.stdout.trim()).toMatch(/^cssert \d+\.\d+\.\d+ \(@cssert\/cli\)$/); + expect((await sb.run(["--help"])).stdout).toContain("@cssert/cli"); }); it("exits 2 for unknown commands", async () => { @@ -76,7 +77,8 @@ describe("cssert check", () => { " ✗ nope templates/bad.html:1:40", " ⚠ text-{{ color }} templates/bad.html:2:22 (dynamic class construction suspected)", "", - " 1 missing, 1 dynamic-suspect · scanned 1 document(s) / 1 stylesheet(s)", + " 1 missing, 1 dynamic-suspect · scanned 1 document(s) / 1 stylesheet(s)" + + " · 1 document(s) still contain unresolved class expressions", "", ].join("\n"), ); @@ -111,7 +113,9 @@ describe("cssert check", () => { className: "nope", kind: "missing", occurrences: [{ path: "templates/bad.html", line: 1, column: 40 }], + sources: ["templates/bad.html"], }); + expect(json.errors).toEqual([]); }); it("writes to --output and reports the location on stderr", async () => { @@ -223,7 +227,7 @@ describe("cssert check with config files", () => { it("accepts --config pointing at a specific file", async () => { sb.write( "conf/custom.json", - JSON.stringify({ css: ["dist/*.css"], html: ["templates/*.html"] }), + JSON.stringify({ css: ["../dist/*.css"], html: ["../templates/*.html"] }), ); expect((await sb.run(["check", "--config", "conf/custom.json"])).code).toBe(1); expect((await sb.run(["check", "--config", "conf/missing.json"])).code).toBe(2); diff --git a/test/cli/coverage.test.ts b/test/cli/coverage.test.ts new file mode 100644 index 0000000..80a2043 --- /dev/null +++ b/test/cli/coverage.test.ts @@ -0,0 +1,175 @@ +import { beforeEach, describe, expect, it } from "vitest"; +import { fixture, type Sandbox, sandbox } from "./helpers.js"; + +const HTML_OK = `
x
\n`; +const HTML_DYNAMIC = `
x
\n`; + +describe("input coverage", () => { + let sb: Sandbox; + beforeEach(() => { + sb = sandbox(); + sb.write("dist/app.css", fixture("tailwind-v4.css")); + sb.write("build/a.html", HTML_OK); + }); + + const base = ["check", "--css", "dist/app.css", "--html", "build/*.html"]; + + it("fails when fewer documents arrived than --min-documents requires", async () => { + expect((await sb.run([...base, "--min-documents", "1"])).code).toBe(0); + const res = await sb.run([...base, "--min-documents", "20"]); + expect(res.code).toBe(1); + expect(res.stdout).toContain("Only 1 HTML document(s) were scanned"); + expect(res.stdout).toContain("input coverage check failed"); + }); + + it("fails when fewer stylesheets arrived than --min-stylesheets requires", async () => { + const res = await sb.run([...base, "--min-stylesheets", "2"]); + expect(res.code).toBe(1); + expect(res.stdout).toContain("Only 1 stylesheet(s) were scanned"); + }); + + it("reports the coverage failure in json, github and sarif", async () => { + const json = JSON.parse( + (await sb.run([...base, "--min-documents", "9", "--format", "json"])).stdout, + ); + expect(json.errors).toHaveLength(1); + + const gh = await sb.run([...base, "--min-documents", "9", "--format", "github"]); + expect(gh.stdout).toContain("::error title=cssert%3A input coverage::"); + + const sarif = JSON.parse( + (await sb.run([...base, "--min-documents", "9", "--format", "sarif"])).stdout, + ); + expect(sarif.runs[0].results[0].ruleId).toBe("cssert/input-coverage"); + }); + + it("rejects a non-integer minimum", async () => { + expect((await sb.run([...base, "--min-documents", "lots"])).code).toBe(2); + sb.write("cssert.config.json", JSON.stringify({ minDocuments: 1.5 })); + expect((await sb.run(base)).code).toBe(2); + }); + + it("counts documents that still contain unresolved class expressions", async () => { + sb.write("build/b.html", HTML_DYNAMIC); + const res = await sb.run(base); + expect(res.code).toBe(0); + expect(res.stdout).toContain("1 document(s) still contain unresolved class expressions"); + const json = JSON.parse((await sb.run([...base, "--format", "json"])).stdout); + expect(json.summary.documentsWithDynamic).toBe(1); + + // A green run still says what it could not verify. + await sb.run(["baseline", "create", ...base.slice(1), "--kind", "all"]); + const green = await sb.run(base); + expect(green.stdout).toContain("✓ no missing classes"); + expect(green.stdout).toContain("1 document(s) still contain unresolved class expressions"); + }); +}); + +describe("negative globs", () => { + it("excludes files matched by a ! pattern from every positive pattern", async () => { + const sb = sandbox(); + sb.write("dist/app.css", fixture("tailwind-v4.css")); + sb.write("build/page.html", `
x
`); + sb.write("build/vendor/page.html", `
x
`); + const base = ["check", "--css", "dist/app.css", "--format", "json"]; + + const all = JSON.parse((await sb.run([...base, "--html", "build/**/*.html"])).stdout); + expect(all.summary.documents).toBe(2); + + const filtered = JSON.parse( + (await sb.run([...base, "--html", "build/**/*.html,!**/vendor/**"])).stdout, + ); + expect(filtered.summary.documents).toBe(1); + expect(filtered.findings.map((f: { className: string }) => f.className)).toEqual(["gone"]); + }); + + it("errors when only negative patterns are given", async () => { + const sb = sandbox(); + sb.write("dist/app.css", fixture("tailwind-v4.css")); + sb.write("build/page.html", `
x
`); + const res = await sb.run(["check", "--css", "dist/app.css", "--html", "!build/**"]); + expect(res.code).toBe(2); + expect(res.stderr).toContain("No html files matched"); + }); +}); + +describe("finding provenance", () => { + let sb: Sandbox; + beforeEach(() => { + sb = sandbox(); + sb.write("dist/app.css", fixture("tailwind-v4.css")); + sb.write("build/rendered/page.html", `
x
`); + sb.write("templates/page.html", `
x
`); + }); + + it("records which input glob each finding came from", async () => { + const res = await sb.run([ + "check", + "--css", + "dist/app.css", + "--html", + "build/rendered/**/*.html,templates/**/*.html", + "--format", + "json", + ]); + expect(res.code).toBe(1); + const bySource = Object.fromEntries( + JSON.parse(res.stdout).findings.map((f: { className: string; sources: string[] }) => [ + f.className, + f.sources, + ]), + ); + // Only in the raw templates: the page that uses it was never rendered. + expect(bySource["template-only"]).toEqual(["templates/**/*.html"]); + expect(bySource["rendered-gone"]).toEqual(["build/rendered/**/*.html", "templates/**/*.html"]); + }); +}); + +describe("hooks and github annotation folding", () => { + let sb: Sandbox; + beforeEach(() => { + sb = sandbox(); + sb.write("dist/app.css", fixture("tailwind-v4.css")); + }); + + it("treats hooks like allow but keeps the intent in the config", async () => { + sb.write("templates/a.html", `
x
`); + const base = ["check", "--css", "dist/app.css", "--html", "templates/a.html"]; + expect((await sb.run(base)).code).toBe(1); + expect((await sb.run([...base, "--hook", "sidebar-collapse-tooltip"])).code).toBe(0); + + sb.write( + "cssert.config.json", + JSON.stringify({ + css: ["dist/app.css"], + html: ["templates/a.html"], + hooks: ["sidebar-collapse-tooltip"], + }), + ); + expect((await sb.run(["check"])).code).toBe(0); + }); + + it("emits one github annotation per class unless --annotate-occurrences", async () => { + for (let i = 0; i < 3; i++) { + sb.write(`templates/p${i}.html`, `
x
`); + } + const base = [ + "check", + "--css", + "dist/app.css", + "--html", + "templates/*.html", + "--format", + "github", + ]; + const folded = (await sb.run(base)).stdout.split("\n").filter((l) => l.startsWith("::")); + expect(folded).toHaveLength(2); + expect(folded[0]).toContain("(and 2 other places)."); + + const all = (await sb.run([...base, "--annotate-occurrences"])).stdout + .split("\n") + .filter((l) => l.startsWith("::")); + expect(all).toHaveLength(6); + expect(all[0]).not.toContain("other places"); + }); +}); diff --git a/test/cli/helpers.ts b/test/cli/helpers.ts index 75e3376..3ea7c42 100644 --- a/test/cli/helpers.ts +++ b/test/cli/helpers.ts @@ -9,6 +9,8 @@ export interface Sandbox { write(relative: string, content: string): string; read(relative: string): string; run(argv: string[], env?: Record): Promise; + /** Run with the working directory set to `relative` inside the sandbox. */ + runIn(relative: string, argv: string[], env?: Record): Promise; } export interface RunResult { @@ -35,11 +37,14 @@ export function sandbox(): Sandbox { read(relative) { return readFileSync(join(dir, relative), "utf8"); }, - async run(argv, env = {}) { + run(argv, env = {}) { + return this.runIn(".", argv, env); + }, + async runIn(relative, argv, env = {}) { let stdout = ""; let stderr = ""; const ctx: CliContext = { - cwd: dir, + cwd: join(dir, relative), stdout: { write(t: string) { stdout += t; diff --git a/test/cli/roots.test.ts b/test/cli/roots.test.ts new file mode 100644 index 0000000..e7a09e4 --- /dev/null +++ b/test/cli/roots.test.ts @@ -0,0 +1,137 @@ +import { mkdirSync } from "node:fs"; +import { join } from "node:path"; +import { beforeEach, describe, expect, it } from "vitest"; +import { resolveRoots } from "../../src/cli/roots.js"; +import { fixture, type Sandbox, sandbox } from "./helpers.js"; + +const HTML_BAD = `
\n

x

\n
\n`; + +describe("resolveRoots", () => { + it("prefers --root, then config.root, then the config directory", () => { + const cwd = "/repo"; + expect(resolveRoots(cwd, "/repo/frontend/cssert.config.mjs", {}, "build")).toEqual({ + config: "/repo/build", + flag: "/repo/build", + }); + expect( + resolveRoots(cwd, "/repo/frontend/cssert.config.mjs", { root: ".." }, undefined), + ).toEqual( + { config: "/repo", flag: "/repo" }, // cwd is /repo here, so both agree + ); + expect( + resolveRoots("/repo/frontend", "/repo/frontend/cssert.config.mjs", { root: ".." }, undefined), + ).toEqual({ config: "/repo", flag: "/repo/frontend" }); + expect(resolveRoots(cwd, "/repo/frontend/cssert.config.mjs", {}, undefined)).toEqual({ + config: "/repo/frontend", + flag: "/repo", + }); + }); + + it("falls back to cwd without a config file or with resolveFrom: cwd", () => { + expect(resolveRoots("/repo", undefined, {}, undefined)).toEqual({ + config: "/repo", + flag: "/repo", + }); + expect( + resolveRoots("/repo", "/repo/frontend/cssert.config.mjs", { resolveFrom: "cwd" }, undefined), + ).toEqual({ config: "/repo", flag: "/repo" }); + }); +}); + +describe("config-relative path resolution", () => { + let sb: Sandbox; + beforeEach(() => { + sb = sandbox(); + sb.write("frontend/dist/assets/app.css", fixture("tailwind-v4.css")); + sb.write("build/rendered/page.html", HTML_BAD); + mkdirSync(join(sb.dir, "frontend/sub"), { recursive: true }); + }); + + it("gives the same result from any working directory", async () => { + sb.write( + "cssert.config.json", + JSON.stringify({ + css: ["frontend/dist/assets/*.css"], + html: ["build/rendered/**/*.html"], + format: "json", + }), + ); + const fromRoot = await sb.run(["check"]); + const fromSub = await sb.runIn("frontend", ["check", "--config", "../cssert.config.json"]); + const fromDeep = await sb.runIn("frontend/sub", [ + "check", + "--config", + "../../cssert.config.json", + ]); + + expect(fromRoot.code).toBe(1); + expect(fromSub.stdout).toBe(fromRoot.stdout); + expect(fromDeep.stdout).toBe(fromRoot.stdout); + expect(JSON.parse(fromRoot.stdout).findings[0].occurrences[0].path).toBe( + "build/rendered/page.html", + ); + }); + + it("resolves config paths against root, and flag globs against cwd", async () => { + sb.write( + "frontend/cssert.config.json", + JSON.stringify({ + root: "..", + css: ["frontend/dist/assets/*.css"], + html: ["build/**/*.html"], + }), + ); + expect((await sb.runIn("frontend", ["check"])).code).toBe(1); + + // A glob typed on the command line still means "relative to where I am". + const flags = await sb.runIn("frontend", [ + "check", + "--css", + "dist/assets/*.css", + "--html", + "../build/rendered/*.html", + ]); + expect(flags.code).toBe(1); + expect(flags.stdout).toContain("../build/rendered/page.html"); + }); + + it("restores cwd-relative resolution with resolveFrom: cwd", async () => { + sb.write( + "frontend/cssert.config.json", + JSON.stringify({ + resolveFrom: "cwd", + css: ["frontend/dist/assets/*.css"], + html: ["build/**/*.html"], + }), + ); + expect((await sb.run(["check", "--config", "frontend/cssert.config.json"])).code).toBe(1); + expect((await sb.runIn("frontend", ["check", "--config", "cssert.config.json"])).code).toBe(2); + expect( + (await sb.run(["check", "--config", "frontend/cssert.config.json", "--root", "."])).code, + ).toBe(1); + }); + + it("writes the baseline next to the config and the budget snapshot too", async () => { + sb.write( + "frontend/cssert.config.json", + JSON.stringify({ + root: "..", + css: ["frontend/dist/assets/*.css"], + html: ["build/**/*.html"], + }), + ); + expect((await sb.runIn("frontend", ["baseline", "create"])).code).toBe(0); + expect(() => sb.read(".cssert/baseline.json")).not.toThrow(); + expect((await sb.runIn("frontend", ["check"])).code).toBe(0); + + expect((await sb.runIn("frontend", ["budget"])).code).toBe(0); + expect(() => sb.read(".cssert/budget.json")).not.toThrow(); + }); + + it("rejects a config whose resolveFrom is not config or cwd", async () => { + sb.write("cssert.config.json", JSON.stringify({ resolveFrom: "nowhere" })); + const res = await sb.run(["check"]); + expect(res.code).toBe(2); + expect(res.stderr).toContain("resolveFrom"); + }); +}); diff --git a/test/report/ci-formats.test.ts b/test/report/ci-formats.test.ts index a98d0c3..5bf2ceb 100644 --- a/test/report/ci-formats.test.ts +++ b/test/report/ci-formats.test.ts @@ -29,38 +29,98 @@ const report: Report = { }, { code: "selector-parse", message: "bad selector" }, ], - stats: { documents: 128, stylesheets: 3, cssClasses: 400, htmlClasses: 120 }, + stats: { + documents: 128, + stylesheets: 3, + cssClasses: 400, + htmlClasses: 120, + documentsWithDynamic: 0, + }, baseline: { path: ".cssert/baseline.json", suppressed: 4 }, }; describe("formatGithub", () => { - it("emits one workflow command per occurrence with escaped properties", () => { + it("emits one annotation per class, anchored at the first occurrence", () => { const out = formatGithub(report).split("\n"); expect(out[0]).toBe( - '::error file=templates/pricing.html,line=42,col=18,title=cssert%3A missing class::Class "lg:my-10" is used here but is not defined in any stylesheet.', + '::error file=templates/pricing.html,line=42,col=18,title=cssert%3A missing class::Class "lg:my-10" is used here but is not defined in any stylesheet (and 1 other place).', ); - expect(out[1]).toContain("line=88,col=10"); - expect(out[2]).toBe( + expect(out[1]).toBe( '::warning file=templates/card.html,line=7,col=22,title=cssert%3A dynamic class::Token "text-{{ color,x }}" contains template syntax and cannot be checked; the class name is probably built dynamically.', ); - expect(out[3]).toBe( + expect(out[2]).toBe( '::error title=cssert%3A missing class::Class "orphan" is used here but is not defined in any stylesheet.', ); - expect(out[4]).toBe( + expect(out[3]).toBe( "::warning file=dist/a.css,line=3,col=1,title=cssert%3A parse warning (css-syntax)::Unclosed block%0Asecond line 100%25", ); - expect(out[5]).toBe("::warning title=cssert%3A parse warning (selector-parse)::bad selector"); - expect(out[6]).toBe( + expect(out[4]).toBe("::warning title=cssert%3A parse warning (selector-parse)::bad selector"); + expect(out[5]).toBe( "cssert: 2 missing, 1 dynamic-suspect, 2 warning(s) · scanned 128 document(s) / 3 stylesheet(s) · 4 suppressed by baseline", ); - expect(out[7]).toBe(""); + expect(out[6]).toBe(""); + }); + + it("emits one annotation per class regardless of how many occurrences it has", () => { + const many = formatGithub({ + ...report, + findings: [ + { + className: "form-control", + kind: "missing", + occurrences: Array.from({ length: 66 }, (_, i) => ({ + path: `templates/p${i}.html`, + line: i + 1, + column: 1, + })), + }, + ], + warnings: [], + }).split("\n"); + expect(many).toHaveLength(3); // one annotation, one summary, trailing "" + expect(many[0]).toContain("file=templates/p0.html,line=1,col=1"); + expect(many[0]).toContain("(and 65 other places)."); + }); + + it("annotates every occurrence with annotateOccurrences", () => { + const out = formatGithub(report, { annotateOccurrences: true }).split("\n"); + expect(out[0]).toBe( + '::error file=templates/pricing.html,line=42,col=18,title=cssert%3A missing class::Class "lg:my-10" is used here but is not defined in any stylesheet.', + ); + expect(out[1]).toContain("line=88,col=10"); + expect(out[1]).not.toContain("other place"); + }); + + it("reports run-level errors and unresolved expressions", () => { + const out = formatGithub({ + findings: [], + warnings: [], + stats: { + documents: 3, + stylesheets: 1, + cssClasses: 1, + htmlClasses: 1, + documentsWithDynamic: 2, + }, + errors: ["Only 3 HTML document(s) were scanned; --min-documents requires at least 20."], + }).split("\n"); + expect(out[0]).toBe( + "::error title=cssert%3A input coverage::Only 3 HTML document(s) were scanned; --min-documents requires at least 20.", + ); + expect(out[1]).toContain("2 document(s) still contain unresolved class expressions"); }); it("prints only the summary for a clean report", () => { const clean = formatGithub({ findings: [], warnings: [], - stats: { documents: 1, stylesheets: 1, cssClasses: 1, htmlClasses: 1 }, + stats: { + documents: 1, + stylesheets: 1, + cssClasses: 1, + htmlClasses: 1, + documentsWithDynamic: 0, + }, }); expect(clean).toBe( "cssert: 0 missing, 0 dynamic-suspect, 0 warning(s) · scanned 1 document(s) / 1 stylesheet(s)\n", @@ -79,6 +139,7 @@ describe("formatSarif", () => { expect(driver?.rules.map((r) => r.id)).toEqual([ SARIF_RULES.missing, SARIF_RULES.dynamic, + SARIF_RULES.coverage, SARIF_RULES.parse, ]); const results = log.runs[0]?.results ?? []; @@ -114,7 +175,13 @@ describe("formatSarif", () => { { className: "x", kind: "missing", occurrences: [{ path: "a.html", line: 0, column: 0 }] }, ], warnings: [{ code: "w", message: "m", path: "b.css" }], - stats: { documents: 1, stylesheets: 1, cssClasses: 1, htmlClasses: 1 }, + stats: { + documents: 1, + stylesheets: 1, + cssClasses: 1, + htmlClasses: 1, + documentsWithDynamic: 0, + }, }); expect(log.runs[0]?.tool.driver.version).toBeUndefined(); expect(log.runs[0]?.results[0]?.locations?.[0]?.physicalLocation.region).toBeUndefined(); diff --git a/test/report/formats.test.ts b/test/report/formats.test.ts index 71b3396..d52986c 100644 --- a/test/report/formats.test.ts +++ b/test/report/formats.test.ts @@ -33,7 +33,13 @@ const report: Report = { }, { code: "selector-parse", message: "bad selector" }, ], - stats: { documents: 128, stylesheets: 3, cssClasses: 400, htmlClasses: 120 }, + stats: { + documents: 128, + stylesheets: 3, + cssClasses: 400, + htmlClasses: 120, + documentsWithDynamic: 0, + }, baseline: { path: ".cssert/baseline.json", suppressed: 4 }, }; @@ -67,7 +73,13 @@ describe("formatHuman", () => { const empty: Report = { findings: [], warnings: [], - stats: { documents: 2, stylesheets: 1, cssClasses: 10, htmlClasses: 5 }, + stats: { + documents: 2, + stylesheets: 1, + cssClasses: 10, + htmlClasses: 5, + documentsWithDynamic: 0, + }, }; expect(formatHuman(empty)).toBe( "\n ✓ no missing classes · scanned 2 document(s) / 1 stylesheet(s)\n", @@ -79,7 +91,13 @@ describe("formatHuman", () => { const dyn: Report = { findings: [{ className: "{{ x }}", kind: "dynamic-suspect", occurrences: [] }], warnings: [], - stats: { documents: 1, stylesheets: 1, cssClasses: 1, htmlClasses: 1 }, + stats: { + documents: 1, + stylesheets: 1, + cssClasses: 1, + htmlClasses: 1, + documentsWithDynamic: 0, + }, }; const out = formatHuman(dyn, { color: true }); expect(out).toContain("\u001b[33m0 missing, 1 dynamic-suspect"); @@ -96,8 +114,10 @@ describe("formatJson", () => { warnings: 2, documents: 128, stylesheets: 3, + documentsWithDynamic: 0, suppressedByBaseline: 4, }); + expect(json.errors).toEqual([]); expect(JSON.parse(formatJson(report))).toEqual(json); expect(formatJson(report).endsWith("\n")).toBe(true); });