Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 5 additions & 24 deletions CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,24 +1,5 @@
# This CODEOWNERS file denotes the project leads
# and encodes their responsibilities for code review.

# Instructions: At a minimum, replace the '@GITHUB_USER_NAME_GOES_HERE'
# here with at least one project lead.

# Lines starting with '#' are comments.
# Each line is a file pattern followed by one or more owners.
# The format is described: https://github.blog/2017-07-06-introducing-code-owners/

# These owners will be the default owners for everything in the repo.
* @daveh-beep


# -----------------------------------------------
# BELOW THIS LINE ARE TEMPLATES, UNUSED
# -----------------------------------------------
# Order is important. The last matching pattern has the most precedence.
# So if a pull request only touches javascript files, only these owners
# will be requested to review.
# *.js @octocat @github/js

# You can also use email addresses if you prefer.
# docs/* docs@example.com
# Upstream Chalkline review routing; not an adopting team's language ownership.
# Adopters: review docs/adoption-checklist.md and replace this owner explicitly.
# Normative protocol/frontmatter changes also require both maintainer roles
# described in GOVERNANCE.md; this default route does not enforce that rule.
* @daveh-beep
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Chalkline helps a team turn its approved voice, terminology, vocabulary, and cha

This open-source project shares reusable methods for maintaining language guidance, not a company's private style guide. It ships fictional examples, a setup protocol, and optional local tools. Your team supplies its own rules and decides who can approve them.

## Status

Experimental protocol and template. Local structural checks and synthetic regressions run in CI; they do not verify writing quality or model adherence. Cross-harness setup parity, real-team setup burden, and ingest limits still need field validation. Cross-repository snapshot projection is a manual review contract, not bundled enforcement. See [sharing readiness and the next experiments](docs/sharing-readiness.md).

## The problem

Your agents write now — support replies, product copy, lifecycle email, release notes. What they know about *your* voice is whatever happened to be in the prompt — and every new prompt, agent file, and skill restates that voice from memory, drifting a little each time. Style guides live in PDFs and wikis that no tool consults. Teams that skip setup blame the tools for output that was never given the chance to be right.
Expand Down Expand Up @@ -48,7 +52,7 @@ Two to four small files plus an optional calibration pair, not a hundred. You co

Target time: under 30 minutes. We're testing that target through real setup experiences.

The upstream `GOVERNANCE.md`, `CODEOWNERS`, `CONTRIBUTING.md`, and `.github/` settings govern contributions to Chalkline itself. When adopting the template, review or replace those settings for your own repository; preserve applicable license and copyright notices. Using Chalkline does not place your language decisions under Block governance or appoint Chalkline maintainers as your reviewers. Agents should propose these changes, never assign owners or remove notices silently.
The upstream `GOVERNANCE.md`, `CODEOWNERS`, `CONTRIBUTING.md`, and `.github/` settings govern contributions to Chalkline itself. When adopting the template, use the [adoption checklist](docs/adoption-checklist.md) to review or replace those settings for your own repository; preserve applicable license and copyright notices. Using Chalkline does not place your language decisions under Block governance or appoint Chalkline maintainers as your reviewers. Agents should propose these changes, never assign owners or remove notices silently.

Before pasting style guides, customer copy, or other source material, read [Safety and privacy](SAFETY.md). Git history can preserve content after you delete it.

Expand Down
16 changes: 16 additions & 0 deletions docs/adoption-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Adopting the template

Complete this review before committing your team's setup. These are repository settings, not new interview questions or language rules. Do not assign owners, remove notices, or publish automatically.

- [ ] Confirm repository visibility and the agent/model provider's data practices. Use only authorized, repository-safe material; see [Safety](../SAFETY.md).
- [ ] Replace or deliberately retain `CODEOWNERS`. Its current owner routes upstream Chalkline reviews, not your team's language decisions.
- [ ] Review `GOVERNANCE.md` and `CONTRIBUTING.md`. Replace upstream processes and links with your own approved ownership and contribution decisions.
- [ ] Review `.github/` issue forms, PR template, workflows, permissions, and environments. In particular, disable or replace the Pages workflow unless you deliberately intend to publish its static site. Check upstream issue labels and contact links.
- [ ] Remove or adapt `website/` if you do not need the upstream explainer. Review upstream GitHub links before using it as your own site.
- [ ] Preserve applicable license and copyright notices. Do not assume a template adoption changes their requirements.
- [ ] Review the generated reference diff, provenance, exact-wording scope, and any separately approved calibration pair. Record unresolved ownership decisions instead of inventing an approver.
- [ ] Decide separately whether to approve references, commit changes, and publish or push the repository.

For use in another repository, follow [PINNING.md](../PINNING.md). A source pin or search index is not a validated snapshot; drafting agents must not fetch upstream instruction files.

This checklist does not verify authority, provider privacy, repository permissions, or that an agent followed the protocol. The adopting owner reviews those decisions.
28 changes: 28 additions & 0 deletions docs/optional-semantic-checks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Optional semantic-check experiment

No TypeSafe/Jev adapter is bundled. This is an experiment plan, not a supported API or an effectiveness claim. Markdown remains canonical; setup, structural checks, and receipt arithmetic need no network or API key.

## Start with fact preservation

Use synthetic supplied facts and candidate drafts. Code extracts bounded candidate spans such as amounts, dates, identifiers, negations, and modal words. An optional semantic model evaluates narrow questions about each fact, such as whether a stated possibility became a promise or a failed request became a confirmed reservation. A generative model still writes copy; the semantic model does not rewrite it.

For retrieval in a larger corpus, first retrieve and filter candidates in code, then evaluate semantic relevance against the supplied task, audience, channel, and locale. Keep no-coverage, unavailable-source, invalid-source, and conflict outcomes distinct. Ranking does not approve a rule or resolve precedence.

## Keep code and people in control

- Code owns parsing, exact comparisons, hashes, arithmetic, explicit scope/precedence, and result routing. Never use a model to sanitize upstream instruction repositories.
- People own authority and all four approval decisions. A semantic answer cannot create a human decision or promote output into guidance.
- Ask independent questions over the same small state together. Do not send the entire repository when only a fact and its surrounding draft are needed.
- Keep raw judgments separate from policy thresholds. TypeSafe Choice/Score confidence and Noul probability are not Chalkline's ordinal 0–3 evidence assessment.
- Expose uncertain answers, errors, timeouts, and skipped checks. Never display an unavailable semantic check as a pass. Requiring a check is an explicit adopting-owner decision.
- Keep credentials server-side or in the local caller's authorized environment. Review provider data practices; avoid body-level debug logging of private inputs. The static site stays offline.

## Acceptance experiment

Predeclare a synthetic suite containing preserved and changed facts, ambiguous dates, unsupported causes, added retry instructions, and irrelevant retrieved passages. Have a separate reviewer label cases without seeing model results. Split tuning and holdout cases; run a deterministic-only control alongside the optional adapter.

Report false negatives and false positives, uncertainty/escalation rate, service failures, cost, end-to-end latency, and review effort. Report the exact model, SDK, questions, candidate spans, and thresholds. Choose thresholds from the consequences and held-out evidence, not a cookbook number. Include adversarially framed source data and option-order checks; typed output does not guarantee truth or injection resistance.

Proceed only if the adapter catches the targeted mutations without unacceptable false alarms or burden on the holdout. It must not change supplied factual strength, suppress blockers, invent evidence, or approve publication. Keep the default tools and CI independent of SDK installation and paid calls.

Before implementation, read the current [TypeSafe docs](https://docs.typesafe.ai/llms.txt), [confidence guidance](https://docs.typesafe.ai/confidence), and [model limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13). Recheck the applicable model's limitations; Jev 1.13 documents injection susceptibility, context distraction, and option-order effects.
31 changes: 31 additions & 0 deletions docs/sharing-readiness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Sharing readiness

Chalkline is an experimental protocol and template for team-owned language guidance. Share it as a method to try, not a verified enforcement or compliance system.

## What has evidence

- Python regression tests cover the local corpus parser, filesystem restrictions, output behavior, and receipt structure/arithmetic.
- CI runs contract-text assertions and validates fictional corpora. Text assertions do not execute agent behavior.
- Website tests exercise fixed fictional tasks and blocked/restored states. The site has no live model, uploads, or telemetry.
- [The July red-team receipt](../checks/RED-TEAM-2026-07-30.md) records single-run observations in three harnesses. It is historical evidence under those conditions, not current setup parity or a reliability estimate.

## What remains unverified

| Work | Current boundary | Next evidence |
|---|---|---|
| Cross-harness setup | Portable instructions, not demonstrated current parity | Run the same synthetic interview in goose, Claude Code, and Codex; [issue #3](https://github.com/block/chalkline/issues/3) |
| Real-team adoption | The under-30-minute setup target is still a target | Recruit a consenting pilot; record task-level elapsed time, stalls, approval burden, and safe gaps; [issue #6](https://github.com/block/chalkline/issues/6) |
| Ingest quality | At most 10 candidate rules per artifact; no measured semantic input budget | Compare predeclared small/large synthetic inputs and unchanged controls; [issue #5](https://github.com/block/chalkline/issues/5) |
| Cross-repository consumption | Manual review and local snapshot contract; no bundled validator/projector | An independently reviewed projector and consuming-harness conformance tests before claiming automated enforcement |
| Website accessibility | DOM-stub tests and historical visual checks | Keyboard, mobile, and screen-reader review; no full accessibility audit is claimed |
| Semantic checking | No Jev integration or automatic fact-preservation check | Evaluate an optional adapter using the [semantic-check experiment](optional-semantic-checks.md) |

## Next execution plan

1. Ship sharing cleanup: current site claims, an adoption checklist, visible structure-only receipt validation, and bounded local corpus reads with synthetic regressions.
2. Run the synthetic setup across the actual target harnesses. Predeclare cases and repetitions, separate reviewer material, preserve unchanged controls, and report failures and unknown settings. Follow [the behavioral-fixture procedure](../checks/behavioral/README.md); do not treat existing unit tests as this evidence.
3. Run a small consenting-team pilot. Keep private artifacts and raw agent sessions out of this public repository. Publish only manually reviewed, sanitized findings or synthetic reproductions.
4. Evaluate optional semantic checks only after the pilot identifies a recurring problem. Keep ordinary setup and local tools dependency-free.
5. Build snapshot projection only if cross-repository use becomes a demonstrated need. Explicit file allowlists and code checks help, but cannot prove arbitrary prose harmless. Keep consuming-owner review and harness access controls.

No change in steps 2–5 is approved by this plan. Language approval, artifact approval, reusable calibration approval, and consumer deployment remain separate decisions. Upstream normative changes require the two maintainer roles in [Governance](../GOVERNANCE.md).
94 changes: 94 additions & 0 deletions tests/test_corpus.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@

import hashlib
import importlib.util
import io
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest import mock

TOOL = Path(__file__).resolve().parents[1] / "tools" / "corpus.py"
# Avoid importlib's bytecode cache: the tool and tests need no disk caches.
Expand Down Expand Up @@ -69,6 +71,98 @@ def test_nested_hashes_and_determinism(self):
first.write_bytes(first.read_bytes() + b"\n")
self.assertNotEqual(before.stdout, self.cli("index").stdout)

def assert_limit_failure(self, constant, limit, message):
with mock.patch.object(corpus, constant, limit):
with self.assertRaisesRegex(corpus.CorpusError, message):
corpus.load_corpus(self.root)
for command in ("check", "index", "search"):
for to_file in (False, True):
with self.subTest(command=command, to_file=to_file):
output = self.root / "result.json"
args = [command, "--root", str(self.root)]
if command == "search":
args += ["payment", "--brand", "other"]
if to_file:
args += ["--output", str(output)]
with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout, \
mock.patch("sys.stderr", new_callable=io.StringIO) as stderr:
self.assertEqual(corpus.main(args), 1)
self.assertEqual(stdout.getvalue(), "")
self.assertIn(message, stderr.getvalue())
self.assertNotIn("Traceback", stderr.getvalue())
self.assertFalse(output.exists())

def test_ingestion_limits_exact_boundary_unchanged_control(self):
self.put("a.md")
self.put("nested/b.md")
self.put("README.md", "ignored")
self.put("notes.txt", "ignored")
expected = corpus.load_corpus(self.root)
size = len(BASE.encode("utf-8"))
with mock.patch.multiple(corpus, MAX_REFERENCE_FILES=2,
MAX_REFERENCE_BYTES=size,
MAX_TOTAL_REFERENCE_BYTES=2 * size,
MAX_DIRECTORY_DEPTH=1, MAX_FILESYSTEM_ENTRIES=5):
self.assertEqual(corpus.load_corpus(self.root), expected)
with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:
self.assertEqual(corpus.main(["index", "--root", str(self.root)]), 0)
self.assertEqual(json.loads(stdout.getvalue())["records"],
[corpus.public_record(record) for record in expected])

def test_reference_file_count_limit(self):
self.put("a.md")
self.put("b.MD")
self.assert_limit_failure("MAX_REFERENCE_FILES", 1, "reference file count limit exceeded")

def test_reference_byte_limit(self):
self.put(text=BASE.replace("Sample", "Café"))
size = len((self.root / "references" / "voice.md").read_bytes())
self.assert_limit_failure("MAX_REFERENCE_BYTES", size - 1, "reference byte limit exceeded")

def test_total_reference_byte_limit(self):
self.put("a.md")
self.put("b.md")
self.assert_limit_failure("MAX_TOTAL_REFERENCE_BYTES", 2 * len(BASE.encode()) - 1,
"total reference byte limit exceeded")

def test_directory_depth_limit_including_ignored_contents(self):
self.put("a.md")
self.put("one/two/notes.txt", "ignored")
self.assert_limit_failure("MAX_DIRECTORY_DEPTH", 1, "directory depth limit exceeded")

def test_filesystem_entry_limit_including_ignored_contents(self):
self.put("a.md")
self.put("nested/README.md", "ignored")
self.put("notes.txt", "ignored")
self.assert_limit_failure("MAX_FILESYSTEM_ENTRIES", 3, "filesystem entry limit exceeded")

def test_reference_read_is_bounded_and_checks_actual_bytes(self):
self.put() # The on-disk size is smaller than the bytes supplied by the stream.
size = len(BASE.encode())
stream = mock.Mock(wraps=io.BytesIO(BASE.encode() + b"extra"))
context = mock.MagicMock()
context.__enter__.return_value = stream
with mock.patch.object(corpus, "MAX_REFERENCE_BYTES", size), \
mock.patch.object(Path, "open", return_value=context), \
self.assertRaisesRegex(corpus.CorpusError, "reference byte limit exceeded"):
corpus.load_corpus(self.root)
stream.read.assert_called_once_with(size + 1)

def test_hash_and_parse_use_same_actual_bytes(self):
self.put()
raw = BASE.replace("Sample", "Café").encode("utf-8")
stream = mock.Mock(wraps=io.BytesIO(raw))
context = mock.MagicMock()
context.__enter__.return_value = stream
with mock.patch.object(corpus, "MAX_REFERENCE_BYTES", len(raw)), \
mock.patch.object(corpus, "MAX_TOTAL_REFERENCE_BYTES", len(raw)), \
mock.patch.object(Path, "open", return_value=context):
record = corpus.load_corpus(self.root)[0]
stream.read.assert_called_once_with(len(raw) + 1)
self.assertEqual(record["metadata"]["title"], "Café")
self.assertEqual(record["bytes"], len(raw))
self.assertEqual(record["sha256"], hashlib.sha256(raw).hexdigest())

def test_search_and_exact_filters(self):
self.put("a.md")
self.put("b.md", BASE.replace('"shared"', '"support"').replace('"should"', '"must"'))
Expand Down
Loading
Loading