diff --git a/.github/workflows/install.yml b/.github/workflows/install.yml new file mode 100644 index 0000000..061eb67 --- /dev/null +++ b/.github/workflows/install.yml @@ -0,0 +1,19 @@ +name: Installer controls +on: + pull_request: + push: + branches: [main] +permissions: + contents: read +jobs: + installer: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + - name: Validate standalone entrypoint and refusal controls + run: | + sh -n install.sh + python3 scripts/test-install-release.py diff --git a/README.md b/README.md index f9d1833..e29872a 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,7 @@ Camayoc logo — loose gray strands enter from the left, the keeper's yupana counting board hangs from the main cord at center, and ordered, colored, knotted quipu pendants emerge on the right

-

camayoc

- -

- 🧶 Agents that check what was already decided, and record new decisions with how they know -

- -

- License: Apache-2.0 - CI - Part of the caboodle stack -

- -[Watch the 90-second native bootstrap recording](docs/demos/bootstrap-v0.1.2/README.md). +# camayoc **Camayoc gives AI coding agents a governed memory: a starter vocabulary for work, decisions and outcomes, loaded into a [quipu](https://github.com/scbrown/quipu) @@ -27,37 +15,20 @@ plain scripts any agent that speaks HTTP can use. The *quipucamayoc* was the Inca official who kept the quipus and certified what was read back out of them. -## Why you would want it - -- **Agents ask before they re-decide.** The skill has them query what was - already decided, and why, before choosing again. -- **Every fact says how it was learned.** A record without an honest - `observed` / `declared` / `inferred` tag is refused at write, so a guess - cannot pass as an observation. -- **It proves its own gate.** Bootstrap sends records that must be refused and - fails loudly if the store accepts one. - -The long form (its place in the stack, what runs today, what it deliberately is -not): [Why camayoc](https://scbrown.github.io/camayoc/why-camayoc.html). - ## Install -As a Claude Code plugin, in Claude Code: - -```text -/plugin marketplace add scbrown/camayoc -/plugin install camayoc@camayoc +```sh +curl -fsSL https://raw.githubusercontent.com/scbrown/camayoc/main/install.sh | sh ``` -From a shell, for any other agent or for scripts: - -```bash -git clone --depth 1 https://github.com/scbrown/camayoc -``` +Python 3.9+, curl, and a Linux or macOS shell. Downloads and verifies the latest source package into `~/.local/share/camayoc/TAG`. +No native executable, server or agent enrollment is installed. -There is nothing else to install first: bootstrap brings its own quipu. +Set `STACK_INSTALL_ROOT` on the receiving shell for another prefix. +Pinned releases, alternatives, configuration and removal are in the +[installation and usage reference](docs/installation-reference.md). -## First success in three commands +## Try it From an empty directory, bootstrap a governed memory and watch it prove its gate: @@ -80,106 +51,14 @@ Decision with no provenance tag. The store refused it. That refusal is the guarantee everything else rests on. Stop the server with `kill "$(cat .quipu/server.pid)"`; the full log is in `bootstrap.log`. -## On your own data - -| you want to | run | -|---|---| -| check whether a question is covered by the competency suite | `python3 camayoc/scripts/competency.py ""` | -| map a question to a stored query with Jev (needs a TypeSafe key) | `python3 camayoc/scripts/competency.py --map ""` | -| preview an advisory ordered-rubric benchmark without a key | `just jev-bench bench/ordinal/na-htm.jsonl --slot ordinal` ([pilot and replay](bench/ordinal/README.md)) | -| see which competency questions have a stored query | `python3 camayoc/scripts/query_coverage.py` | -| check a proposal against decisions already made | `python3 camayoc/scripts/settled_decisions.py "" --declared ` | -| see the quarantine planes and their labels | `python3 camayoc/scripts/planes.py status` | - -Every script: [Reference](https://scbrown.github.io/camayoc/reference.html). - -## Wire it into your agent - -With the plugin installed, run this once in Claude Code: - -```text -/camayoc:bootstrap -``` - -It does what the shell bootstrap does, then offers the rest of the stack. -From then on every session opens with a memory status line (`ACTIVE`, -`reachable but gate not proven`, or `unreachable`), and the skill teaches the -agent to query first, record at the moment, and tag honestly. Without the -plugin, `bash camayoc/scripts/bootstrap.sh --with-claude-hooks` adds the same -status hook to `.claude/settings.json`. The plugin also declares a `jev` MCP -server for typed decisions. Details: [Using it with agents](https://scbrown.github.io/camayoc/agents.html). - -### Typed decisions with jev-mcp - -The plugin registers the stdio server in -[scripts/jev_mcp.py](scripts/jev_mcp.py). It exposes these tools: +## Next steps -| Tool | Input and result | -| --- | --- | -| `jev_noul` | `state`, `instructions`; returns a yes/no probability. Optional `true` and `false` describe the criteria. | -| `jev_choice` | `state`, `instructions`, `criteria` (option ID to text); returns a choice and probabilities. | -| `jev_score` | `state`, `instructions`, `levels` (2–10 ordered strings); returns an ordered-level score. | -| `map_question` | `question`; reports a competency-question match and its stored-query coverage, or `abstained` / `human_reads`. | -| `jev_dry_run` | `kind` (`noul`, `choice` or `score`) and that tool's inputs; returns the request without sending it. | +- [Tambo](https://github.com/intigraph/tambo): guided installation of the complete stack. +- [Detailed examples and agent setup](docs/installation-reference.md). -For an agent that reads `.mcp.json`, register the server directly: +**Load this repository’s graph:** public repository-pendant assets have not yet been verified. When a release supplies the archive, checksum and provenance, follow the [verified import workflow](https://scbrown.github.io/quipu/sharing/index.html): extract the share directory and use `quipu import SHARE_DIR --db repository.db`. Import stages knowledge; ROOT stays unchanged. -```json -{ - "mcpServers": { - "jev": { - "command": "python3", - "args": ["/path/to/camayoc/scripts/jev_mcp.py"] - } - } -} -``` - -Live calls use `TYPESAFE_API_KEY`, then the file named by -`TYPESAFE_API_KEY_FILE`, then the default in -[`jev.resolve_key`](scripts/jev.py). Keep a key file private (`0600`); -do not put a key in the registration file. Dry runs need no key. A missing -or malformed key produces a readable tool error, with no lexical fallback. - -Choice adds a none-of-these option by default and reports `chose_none`; -setting `none_text` to `""` deliberately removes that escape hatch. -Treat decisions as **inferred**, retain their request, usage and model, and -check confidence before acting. The server does not retry failed calls. -Usage is logged to `JEV_USAGE_LOG`, otherwise -`SHANTY_ROOT/jev-usage.jsonl`, otherwise the local state path declared in -`scripts/jev_mcp.py`. Logging failure does not retry a decision. - -## Deployment events - -Camayoc records deployments from the tools that make them, as CDEvents mapped -to `DeploymentEvent`. Each emitter reports what the tool itself says -happened, including failures that a surrounding job reports as green. - -| Tool | How | Standard | -| --- | --- | --- | -| GitHub or Forgejo Actions | [reusable Action](actions/deployment-event/README.md) | CDEvents `pipelineRun.finished` | -| Ansible | [callback plugin](emitters/ansible/README.md) | CDEvents `pipelineRun.finished`, one per host | -| Terraform | [`apply -json` wrapper](emitters/terraform/README.md) | CDEvents `pipelineRun.finished` | - -## Before you start - -**Platforms.** Bootstrap downloads a quipu release on Linux x86_64. On other -platforms it builds quipu with `cargo`, or uses a `quipu-server` already on -your `PATH`. The scripts need `bash`, `curl` and `python3`. - -**Exit codes.** Bootstrap exits 1 if nothing could be installed, 2 if the store -accepted a record it must refuse, and 3 if no verdict could be reached. See -[Getting started](https://scbrown.github.io/camayoc/getting-started.html#when-something-fails). - -## What's next - -- [The camayoc book](https://scbrown.github.io/camayoc/): getting started, agents, reference and the design notes, in reading order -- [Docs map](https://scbrown.github.io/camayoc/docs-map.html): every document in this repository, routed -- [Ingress](https://scbrown.github.io/camayoc/design/ingress.html): how knowledge earns its way into the graph - -## 🧺 The stack - -Caboodle installs these together and proves each one works; every tool also stands alone. +## The stack @@ -205,57 +84,4 @@ when. -## Contributing - -```bash -just test -just check # the local quality gate -``` - -## 📜 License - -[Apache-2.0](LICENSE) - -### Incremental verdict adapters - -`blocked_by.py` and `review_due.py` also accept `--describe` and `--changes` -for Chaski's incremental protocol. `--describe` prints watched predicates, type -IRIs and trusted planes without querying the store. `--changes` consumes one -JSON request with `now` and either `discover: true`, `route: [fact changes]`, or -`items: [local keys]`. Responses carry protocol `version: 1` and respectively -`items`, `items`, or `scope`/`records`/`next_checks`. - -Routing follows changed observations to their WorkItems, then reverse blocker -edges to dependent work. Review routing follows verification links, including -retracted `old_value` links. Existing graph-plane and legacy/public vocabulary -rules apply unchanged. Discovery reads the candidate catalogue only; Chaski -queues bounded evaluation batches instead of scanning verdicts in one process. -The command budgets graph reads at eight per second and caches identical reads -within a request. The ordinary full-evaluation CLI remains the parity oracle. - -Future review ages and date blockers supply exact deadlines. Only external -probes and arbitrary resolution queries retain individual 15-minute checks, -because their dependencies cannot be enumerated from graph changes. Missing -facts and unanchored ages remain UNKNOWN and await a relevant change; they do -not cause a repeated population scan. Transport failures fail the invocation, -so the consumer retains pending work. This protocol performs no graph writes. - -### Reflection cycle and lane expiry - -Tracker ingress classifies a `dream-cycle` label as `DreamCycle`. Its ordinary -idle limit follows tracker activity. A crew cycle also has the `dream` label and -the controller's `DREAM consolidate:` or `DREAM propose:` title. Its creation records an immutable -observation of the reflection lane, typed `Service`, with kind `DreamLane` and a -12-hour idle limit. The lane clock uses cycle creation time; later comments, -completion, retries and out-of-order ingress cannot postpone or rewind it. - -The existing change adapter subscribes to kind changes, observations and age -attributes. Persisted deadlines drive lapses without adding a periodic scanner. -Review verdicts carry a unique kind as receiver context; ambiguous kinds omit it -and retain the receiver's ordinary policy. Routing severity belongs to reviewed -receiver configuration. Load the expanded core shape set before enabling ingress -that writes `DreamLane`; this feature adds a kind value, not a new RDF class. - -Lane signals measure absence of cycle creation, not successful completion or -available capacity. Shantytown retains its capacity gates and rotates past stale -queued cycles; this projection never closes a claimed item or starts a crew pane. +[License](LICENSE). diff --git a/docs/installation-reference.md b/docs/installation-reference.md new file mode 100644 index 0000000..48481ce --- /dev/null +++ b/docs/installation-reference.md @@ -0,0 +1,237 @@ +

+ Camayoc logo — loose gray strands enter from the left, the keeper's yupana counting board hangs from the main cord at center, and ordered, colored, knotted quipu pendants emerge on the right +

+ +

camayoc

+ +

+ 🧶 Agents that check what was already decided, and record new decisions with how they know +

+ +

+ License: Apache-2.0 + CI + Part of the caboodle stack +

+ +[Watch the 90-second native bootstrap recording](https://github.com/scbrown/camayoc/blob/main/docs/demos/bootstrap-v0.1.2/README.md). + +**Camayoc gives AI coding agents a governed memory: a starter vocabulary for +work, decisions and outcomes, loaded into a [quipu](https://github.com/scbrown/quipu) +knowledge graph that refuses any record that does not say how it was learned +(observed, declared or inferred).** It is for teams whose agents keep +re-deciding things a past session already settled. It comes as a Claude Code +plugin (a skill, a session-start status hook and a bootstrap command), and as +plain scripts any agent that speaks HTTP can use. + +The *quipucamayoc* was the Inca official who kept the quipus and certified what +was read back out of them. + +## Why you would want it + +- **Agents ask before they re-decide.** The skill has them query what was + already decided, and why, before choosing again. +- **Every fact says how it was learned.** A record without an honest + `observed` / `declared` / `inferred` tag is refused at write, so a guess + cannot pass as an observation. +- **It proves its own gate.** Bootstrap sends records that must be refused and + fails loudly if the store accepts one. + +The long form (its place in the stack, what runs today, what it deliberately is +not): [Why camayoc](https://scbrown.github.io/camayoc/why-camayoc.html). + +## Install + +As a Claude Code plugin, in Claude Code: + +```text +/plugin marketplace add scbrown/camayoc +/plugin install camayoc@camayoc +``` + +From a shell, for any other agent or for scripts: + +```bash +git clone --depth 1 https://github.com/scbrown/camayoc +``` + +There is nothing else to install first: bootstrap brings its own quipu. + +## First success in three commands + +From an empty directory, bootstrap a governed memory and watch it prove its gate: + +```bash +git clone -q --depth 1 https://github.com/scbrown/camayoc +QUIPU_SERVER=http://127.0.0.1:3030 bash camayoc/scripts/bootstrap.sh > bootstrap.log 2>&1 +grep -E '^(ontology|shapes|gate: PROVEN — an untagged|camayoc:)' bootstrap.log +``` + +```text +ontology: loaded (core.ttl) +shapes: loaded (camayoc-core) +gate: PROVEN — an untagged Decision refused, as it must be. +camayoc: governed memory ready. Query first; record at the moment; tag honestly. +``` + +With no quipu on your machine, bootstrap downloaded one (sha256-checked), +started it, loaded the vocabulary and shapes, and then tried to write a +Decision with no provenance tag. The store refused it. That refusal is the +guarantee everything else rests on. Stop the server with +`kill "$(cat .quipu/server.pid)"`; the full log is in `bootstrap.log`. + +## On your own data + +| you want to | run | +|---|---| +| check whether a question is covered by the competency suite | `python3 camayoc/scripts/competency.py ""` | +| map a question to a stored query with Jev (needs a TypeSafe key) | `python3 camayoc/scripts/competency.py --map ""` | +| preview an advisory ordered-rubric benchmark without a key | `just jev-bench bench/ordinal/na-htm.jsonl --slot ordinal` ([pilot and replay](https://github.com/scbrown/camayoc/blob/main/bench/ordinal/README.md)) | +| see which competency questions have a stored query | `python3 camayoc/scripts/query_coverage.py` | +| check a proposal against decisions already made | `python3 camayoc/scripts/settled_decisions.py "" --declared ` | +| see the quarantine planes and their labels | `python3 camayoc/scripts/planes.py status` | + +Every script: [Reference](https://scbrown.github.io/camayoc/reference.html). + +## Wire it into your agent + +With the plugin installed, run this once in Claude Code: + +```text +/camayoc:bootstrap +``` + +It does what the shell bootstrap does, then offers the rest of the stack. +From then on every session opens with a memory status line (`ACTIVE`, +`reachable but gate not proven`, or `unreachable`), and the skill teaches the +agent to query first, record at the moment, and tag honestly. Without the +plugin, `bash camayoc/scripts/bootstrap.sh --with-claude-hooks` adds the same +status hook to `.claude/settings.json`. The plugin also declares a `jev` MCP +server for typed decisions. Details: [Using it with agents](https://scbrown.github.io/camayoc/agents.html). + +### Typed decisions with jev-mcp + +The plugin registers the stdio server in +[scripts/jev_mcp.py](https://github.com/scbrown/camayoc/blob/main/scripts/jev_mcp.py). It exposes these tools: + +| Tool | Input and result | +| --- | --- | +| `jev_noul` | `state`, `instructions`; returns a yes/no probability. Optional `true` and `false` describe the criteria. | +| `jev_choice` | `state`, `instructions`, `criteria` (option ID to text); returns a choice and probabilities. | +| `jev_score` | `state`, `instructions`, `levels` (2–10 ordered strings); returns an ordered-level score. | +| `map_question` | `question`; reports a competency-question match and its stored-query coverage, or `abstained` / `human_reads`. | +| `jev_dry_run` | `kind` (`noul`, `choice` or `score`) and that tool's inputs; returns the request without sending it. | + +For an agent that reads `.mcp.json`, register the server directly: + +```json +{ + "mcpServers": { + "jev": { + "command": "python3", + "args": ["/path/to/camayoc/scripts/jev_mcp.py"] + } + } +} +``` + +Live calls use `TYPESAFE_API_KEY`, then the file named by +`TYPESAFE_API_KEY_FILE`, then the default in +[`jev.resolve_key`](https://github.com/scbrown/camayoc/blob/main/scripts/jev.py). Keep a key file private (`0600`); +do not put a key in the registration file. Dry runs need no key. A missing +or malformed key produces a readable tool error, with no lexical fallback. + +Choice adds a none-of-these option by default and reports `chose_none`; +setting `none_text` to `""` deliberately removes that escape hatch. +Treat decisions as **inferred**, retain their request, usage and model, and +check confidence before acting. The server does not retry failed calls. +Usage is logged to `JEV_USAGE_LOG`, otherwise +`SHANTY_ROOT/jev-usage.jsonl`, otherwise the local state path declared in +`scripts/jev_mcp.py`. Logging failure does not retry a decision. + +## Deployment events + +Camayoc records deployments from the tools that make them, as CDEvents mapped +to `DeploymentEvent`. Each emitter reports what the tool itself says +happened, including failures that a surrounding job reports as green. + +| Tool | How | Standard | +| --- | --- | --- | +| GitHub or Forgejo Actions | [reusable Action](https://github.com/scbrown/camayoc/blob/main/actions/deployment-event/README.md) | CDEvents `pipelineRun.finished` | +| Ansible | [callback plugin](https://github.com/scbrown/camayoc/blob/main/emitters/ansible/README.md) | CDEvents `pipelineRun.finished`, one per host | +| Terraform | [`apply -json` wrapper](https://github.com/scbrown/camayoc/blob/main/emitters/terraform/README.md) | CDEvents `pipelineRun.finished` | + +## Before you start + +**Platforms.** Bootstrap downloads a quipu release on Linux x86_64. On other +platforms it builds quipu with `cargo`, or uses a `quipu-server` already on +your `PATH`. The scripts need `bash`, `curl` and `python3`. + +**Exit codes.** Bootstrap exits 1 if nothing could be installed, 2 if the store +accepted a record it must refuse, and 3 if no verdict could be reached. See +[Getting started](https://scbrown.github.io/camayoc/getting-started.html#when-something-fails). + +## What's next + +- [The camayoc book](https://scbrown.github.io/camayoc/): getting started, agents, reference and the design notes, in reading order +- [Docs map](https://scbrown.github.io/camayoc/docs-map.html): every document in this repository, routed +- [Ingress](https://scbrown.github.io/camayoc/design/ingress.html): how knowledge earns its way into the graph + +## 🧺 The stack + +Caboodle installs these together and proves each one works; every tool also stands alone. + +## Contributing + +```bash +just test +just check # the local quality gate +``` + +## 📜 License + +[Apache-2.0](https://github.com/scbrown/camayoc/blob/main/LICENSE) + +### Incremental verdict adapters + +`blocked_by.py` and `review_due.py` also accept `--describe` and `--changes` +for Chaski's incremental protocol. `--describe` prints watched predicates, type +IRIs and trusted planes without querying the store. `--changes` consumes one +JSON request with `now` and either `discover: true`, `route: [fact changes]`, or +`items: [local keys]`. Responses carry protocol `version: 1` and respectively +`items`, `items`, or `scope`/`records`/`next_checks`. + +Routing follows changed observations to their WorkItems, then reverse blocker +edges to dependent work. Review routing follows verification links, including +retracted `old_value` links. Existing graph-plane and legacy/public vocabulary +rules apply unchanged. Discovery reads the candidate catalogue only; Chaski +queues bounded evaluation batches instead of scanning verdicts in one process. +The command budgets graph reads at eight per second and caches identical reads +within a request. The ordinary full-evaluation CLI remains the parity oracle. + +Future review ages and date blockers supply exact deadlines. Only external +probes and arbitrary resolution queries retain individual 15-minute checks, +because their dependencies cannot be enumerated from graph changes. Missing +facts and unanchored ages remain UNKNOWN and await a relevant change; they do +not cause a repeated population scan. Transport failures fail the invocation, +so the consumer retains pending work. This protocol performs no graph writes. + +### Reflection cycle and lane expiry + +Tracker ingress classifies a `dream-cycle` label as `DreamCycle`. Its ordinary +idle limit follows tracker activity. A crew cycle also has the `dream` label and +the controller's `DREAM consolidate:` or `DREAM propose:` title. Its creation records an immutable +observation of the reflection lane, typed `Service`, with kind `DreamLane` and a +12-hour idle limit. The lane clock uses cycle creation time; later comments, +completion, retries and out-of-order ingress cannot postpone or rewind it. + +The existing change adapter subscribes to kind changes, observations and age +attributes. Persisted deadlines drive lapses without adding a periodic scanner. +Review verdicts carry a unique kind as receiver context; ambiguous kinds omit it +and retain the receiver's ordinary policy. Routing severity belongs to reviewed +receiver configuration. Load the expanded core shape set before enabling ingress +that writes `DreamLane`; this feature adds a kind value, not a new RDF class. + +Lane signals measure absence of cycle creation, not successful completion or +available capacity. Shantytown retains its capacity gates and rotates past stale +queued cycles; this projection never closes a claimed item or starts a crew pane. diff --git a/install.sh b/install.sh new file mode 100755 index 0000000..d9def2d --- /dev/null +++ b/install.sh @@ -0,0 +1,247 @@ +#!/bin/sh +# Verify the published package before installing; no service or enrollment action. +set -eu +exec python3 - "$@" <<'PYINSTALL' +import hashlib +import json +import os +from pathlib import Path, PurePosixPath +import platform +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile +import time +from urllib.error import HTTPError, URLError +from urllib.request import Request, urlopen + + + + +def fetch(url, destination): + for attempt in range(3): + try: + with urlopen(Request(url, headers={'User-Agent': CONFIG['repo'] + '-installer'}), timeout=60) as response: + with destination.open('wb') as output: + shutil.copyfileobj(response, output) + return + except (HTTPError, URLError, TimeoutError) as error: + transient = not isinstance(error, HTTPError) or error.code in (403, 429, 500, 502, 503, 504) + if attempt == 2 or not transient: + raise + time.sleep(attempt + 1) + + +def extract(archive, destination): + with tarfile.open(archive, 'r:gz') as bundle: + seen, links = set(), [] + total = 0 + for member in bundle: + name = PurePosixPath(member.name) + if name.is_absolute() or '..' in name.parts or str(name) in seen: + raise ValueError('unsafe or duplicate archive path') + seen.add(str(name)) + target = destination.joinpath(*name.parts) + target.parent.mkdir(parents=True, exist_ok=True) + if member.isdir(): + target.mkdir(exist_ok=True) + elif member.isfile(): + total += member.size + if total > 1024 * 1024 * 1024: + raise ValueError('release exceeds 1 GiB unpacked limit') + source = bundle.extractfile(member) + if source is None: + raise ValueError('missing archive bytes') + with source, target.open('xb') as output: + shutil.copyfileobj(source, output) + target.chmod(0o755 if member.mode & 0o111 else 0o644) + elif member.issym(): + link = PurePosixPath(member.linkname) + if link.is_absolute() or '..' in link.parts: + raise ValueError('unsafe archive link') + links.append((target, member.linkname)) + else: + raise ValueError('unsupported archive member') + for target, link in links: + target.symlink_to(link) + target.resolve(strict=True).relative_to(destination.resolve()) + + +CONFIG = {'kind': 'source', 'assets': ['camayoc-{tag}.tar.gz'], 'checksum': 'SHA256SUMS', 'repo': 'camayoc'} + + +def bundle_identity(root): + result = {} + for path in root.rglob('*'): + relative = str(path.relative_to(root)) + if relative == '.release.json': + continue + if path.is_symlink(): + result[relative] = ('link', os.readlink(path)) + elif path.is_file(): + result[relative] = ('file', path.stat().st_mode & 0o777, + hashlib.sha256(path.read_bytes()).hexdigest()) + elif path.is_dir(): + result[relative] = ('directory',) + else: + raise ValueError('unexpected installed bundle entry') + return result + + +def install(): + if len(sys.argv) > 1 or platform.system() not in ('Linux', 'Darwin'): + raise ValueError('use a Linux or macOS shell; configure STACK_INSTALL_ROOT through the environment') + if CONFIG['kind'] == 'wheel' and sys.version_info < (3, 11): + raise ValueError('Python 3.11 or newer is required') + with tempfile.TemporaryDirectory(prefix=CONFIG['repo'] + '-release-') as temporary: + work = Path(temporary) + tag = os.environ.get('STACK_VERSION') + if not tag: + metadata = work / 'latest.json' + fetch('https://api.github.com/repos/scbrown/' + CONFIG['repo'] + '/releases/latest', metadata) + release = json.loads(metadata.read_text()) + if release.get('draft') or release.get('prerelease'): + raise ValueError('latest release is not stable') + tag = release['tag_name'] + if not re.fullmatch(r'v[0-9]+\.[0-9]+\.[0-9]+', tag): + raise ValueError('unsupported release tag') + version = tag[1:] + base = os.environ.get('STACK_RELEASE_BASE_URL', 'https://github.com/scbrown/' + CONFIG['repo'] + '/releases/download/' + tag) + manifest = work / 'checksums' + fetch(base + '/' + CONFIG['checksum'], manifest) + names = [name.format(tag=tag, version=version) for name in CONFIG['assets']] + digests = {} + for name in names: + matches = [fields[0].lower() for line in manifest.read_text().splitlines() + if len(fields := line.split()) == 2 and fields[1].lstrip('*') == name] + if len(matches) != 1 or not re.fullmatch('[0-9a-f]{64}', matches[0]): + raise ValueError('missing or ambiguous checksum for ' + name) + fetch(base + '/' + name, work / name) + digest = hashlib.sha256((work / name).read_bytes()).hexdigest() + if digest != matches[0]: + raise ValueError('checksum mismatch: ' + name) + digests[name] = digest + prefix = Path(os.environ.get('STACK_INSTALL_ROOT', str(Path.home() / '.local'))).expanduser().absolute() + releases = prefix / 'share' / CONFIG['repo'] + releases.mkdir(parents=True, exist_ok=True) + lock = releases / '.install-lock' + lock.mkdir() + destination = releases / tag + identity = {'tag': tag, 'assets': digests} + created = False + try: + if destination.is_symlink(): + raise ValueError('release destination is a symlink') + if destination.exists(): + if json.loads((destination / '.release.json').read_text()) != identity: + raise ValueError('existing release identity differs; previous install preserved') + if CONFIG['kind'] == 'source': + check = work / 'source-check' + check.mkdir() + extract(work / names[0], check) + if bundle_identity(destination) != bundle_identity(check / ('camayoc-' + tag)): + raise ValueError('installed source bytes differ') + elif CONFIG['kind'] == 'data': + for name in names: + if hashlib.sha256((destination / name).read_bytes()).hexdigest() != digests[name]: + raise ValueError('installed data bytes differ') + else: + if CONFIG['kind'] == 'wheel': + # The venv is created at its final location so console-script + # shebangs remain valid. No command is published before smoke. + destination.mkdir() + created = True + subprocess.run([sys.executable, '-m', 'venv', str(destination)], check=True, timeout=120) + python = destination / 'bin/python' + subprocess.run([str(python), '-m', 'pip', '--isolated', '--disable-pip-version-check', + 'install', '--no-input', str(work / names[0])], check=True, timeout=300) + actual = subprocess.check_output([str(python), '-c', + 'from importlib.metadata import version; print(version(' + repr(CONFIG['package']) + '))'], text=True, timeout=20).strip() + if actual != version: + raise ValueError('installed package version mismatch') + for command in CONFIG['commands']: + if not os.access(destination / 'bin' / command, os.X_OK): + raise ValueError('missing console entrypoint: ' + command) + elif CONFIG['kind'] == 'source': + unpacked = work / 'source' + unpacked.mkdir() + extract(work / names[0], unpacked) + source = unpacked / ('camayoc-' + tag) + actual = json.loads((source / '.claude-plugin/plugin.json').read_text())['version'] + if actual != version: + raise ValueError('source package version mismatch') + # A failed copy must never leave an unmarked final release. + # TemporaryDirectory removes partial copies before a retry. + with tempfile.TemporaryDirectory(prefix='.candidate-', dir=releases) as candidate: + staged = Path(candidate) / 'bundle' + shutil.copytree(source, staged, symlinks=True) + staged.rename(destination) + created = True + else: + destination.mkdir() + created = True + for name in names: + data = (work / name).read_bytes() + if not data.strip(): + raise ValueError('empty vocabulary asset') + shutil.copyfile(work / name, destination / name) + (destination / '.release.json').write_text(json.dumps(identity, sort_keys=True) + '\n') + if CONFIG['kind'] == 'wheel': + # Pip's RECORD binds installed package/entrypoint bytes. Validate + # it on both first install and reuse; a stale marker is not proof. + code = """from importlib.metadata import distribution +from pathlib import Path +import base64, hashlib, sys +root = Path(sys.prefix).resolve() +dist = distribution(sys.argv[1]) +assert dist.version == sys.argv[2], 'installed package version mismatch' +for entry in dist.files or []: + if entry.hash is None: + continue + path = Path(dist.locate_file(entry)).resolve() + path.relative_to(root) + assert path.is_file(), 'installed package entry is not a regular file' + actual = base64.urlsafe_b64encode(hashlib.new(entry.hash.mode, path.read_bytes()).digest()).decode().rstrip('=') + assert actual == entry.hash.value, 'installed package file hash mismatch' +""" + subprocess.run([str(destination / 'bin/python'), '-c', code, + CONFIG['package'], version], check=True, timeout=30) + if CONFIG['kind'] == 'wheel': + binaries = prefix / 'bin' + binaries.mkdir(parents=True, exist_ok=True) + for command in CONFIG['commands']: + if (binaries / command).is_dir(): + raise ValueError('command destination is a directory') + for command in CONFIG['commands']: + fd, staged = tempfile.mkstemp(prefix='.' + command + '-', dir=binaries) + os.close(fd) + os.unlink(staged) + try: + os.symlink(destination / 'bin' / command, staged) + os.replace(staged, binaries / command) + finally: + if os.path.lexists(staged): + os.unlink(staged) + print('Installed verified ' + CONFIG['repo'] + ' ' + version + ' at ' + str(destination)) + if CONFIG['kind'] == 'source': + print('Source package only. Bootstrap and agent enrollment are separate explicit actions.') + elif CONFIG['kind'] == 'data': + print('Vocabulary and shapes only. Nothing was loaded into a running graph.') + else: + print('Add ' + str(prefix / 'bin') + ' to PATH. Previous release directories are retained.') + except BaseException: + if created: + shutil.rmtree(destination) + raise + finally: + lock.rmdir() + + +try: + install() +except (OSError, ValueError, KeyError, subprocess.SubprocessError, tarfile.TarError) as error: + print('Install refused: ' + str(error), file=sys.stderr) + sys.exit(1) +PYINSTALL diff --git a/justfile b/justfile index 203f96e..a6e1943 100644 --- a/justfile +++ b/justfile @@ -68,6 +68,11 @@ query-coverage *args: examples-qpack *args: @python3 scripts/examples_qpack.py {{args}} +# Verify the standalone installer using private release fixtures. +installer-check: + sh -n install.sh + python3 scripts/test-install-release.py + # Copied pointer-only repository-pendant controls; no publication or live store. repository-pendant-tests: python3 scripts/test-repository-knowledge.py diff --git a/scripts/test-install-release.py b/scripts/test-install-release.py new file mode 100644 index 0000000..c57f9a9 --- /dev/null +++ b/scripts/test-install-release.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Offline package fixtures exercise the real standalone installer.""" +from contextlib import redirect_stderr, redirect_stdout +import hashlib +import io +import json +import os +from pathlib import Path +import subprocess +import tarfile +import tempfile +import unittest +from urllib.request import urlopen as real_urlopen +from unittest.mock import patch +import zipfile + +ROOT = Path(__file__).resolve().parents[1] +CONFIG = {'kind': 'source', 'assets': ['camayoc-{tag}.tar.gz'], 'checksum': 'SHA256SUMS', 'repo': 'camayoc'} +TAG = 'v1.2.3' +VERSION = '1.2.3' + + +class PackageInstall(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix='package-install-test-') + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.release = self.root / 'release' + self.release.mkdir() + self.prefix = self.root / 'prefix' + (self.prefix / 'bin').mkdir(parents=True) + self.previous = self.prefix / 'bin' / CONFIG.get('commands', ['prior'])[0] + self.previous.write_text('previous bytes') + self.env = dict(os.environ, HOME=str(self.root / 'home'), STACK_VERSION=TAG, + STACK_INSTALL_ROOT=str(self.prefix), STACK_RELEASE_BASE_URL=self.release.as_uri()) + self.names = [name.format(tag=TAG, version=VERSION) for name in CONFIG['assets']] + self.fixture() + self.latest = {"tag_name": TAG, "draft": False, "prerelease": False} + + def fixture(self): + for name in self.names: + path = self.release / name + if CONFIG['kind'] == 'wheel': + distribution = CONFIG['package'].replace('-', '_') + metadata = distribution + '-' + VERSION + '.dist-info/' + with zipfile.ZipFile(path, 'w') as wheel: + wheel.writestr('fixture_cli.py', 'def main():\n print("fixture ' + VERSION + '")\n') + wheel.writestr(metadata + 'METADATA', 'Metadata-Version: 2.1\nName: ' + CONFIG['package'] + '\nVersion: ' + VERSION + '\n') + wheel.writestr(metadata + 'WHEEL', 'Wheel-Version: 1.0\nGenerator: fixture\nRoot-Is-Purelib: true\nTag: py3-none-any\n') + wheel.writestr(metadata + 'entry_points.txt', '[console_scripts]\n' + '\n'.join(command + ' = fixture_cli:main' for command in CONFIG['commands'])) + wheel.writestr(metadata + 'RECORD', '') + elif CONFIG['kind'] == 'source': + data = json.dumps({'name': 'camayoc', 'version': VERSION}).encode() + with tarfile.open(path, 'w:gz') as bundle: + member = tarfile.TarInfo('camayoc-' + TAG + '/.claude-plugin/plugin.json') + member.size, member.mode = len(data), 0o644 + bundle.addfile(member, io.BytesIO(data)) + else: + path.write_text('@prefix example: .\nexample:Thing a example:Class .\n') + manifest = ''.join(hashlib.sha256((self.release / name).read_bytes()).hexdigest() + ' ' + name + '\n' for name in self.names) + (self.release / CONFIG['checksum']).write_text(manifest) + + def open_release(self, request, **kwargs): + if request.full_url.startswith('https://api.github.com/repos/scbrown/'): + return io.BytesIO(json.dumps(self.latest).encode()) + return real_urlopen(request, **kwargs) + + def run_installer(self, system='Linux'): + script = (ROOT / 'install.sh').read_text().split("<<'PYINSTALL'\n", 1)[1].rsplit('PYINSTALL', 1)[0] + stdout, stderr = io.StringIO(), io.StringIO() + status = 0 + with patch.dict(os.environ, self.env, clear=True), patch('sys.argv', ['install.sh']), patch('urllib.request.urlopen', side_effect=self.open_release), \ + patch('platform.system', return_value=system), redirect_stdout(stdout), redirect_stderr(stderr): + try: + exec(compile(script, str(ROOT / 'install.sh'), 'exec'), {'__name__': '__main__'}) + except SystemExit as error: + status = error.code + return status, stdout.getvalue(), stderr.getvalue() + + def assert_preserved(self, result): + self.assertNotEqual(result[0], 0, result) + self.assertEqual(self.previous.read_text(), 'previous bytes') + self.assertFalse(self.previous.is_symlink()) + self.assertNotIn('Installed verified', result[1]) + + def test_actual_installer_installs_verified_package(self): + result = self.run_installer() + self.assertEqual(result[0], 0, result) + destination = self.prefix / 'share' / CONFIG['repo'] / TAG + self.assertEqual(json.loads((destination / '.release.json').read_text())['tag'], TAG) + if CONFIG['kind'] == 'wheel': + for command in CONFIG['commands']: + self.assertIn(VERSION, subprocess.check_output([str(self.prefix / 'bin' / command)], text=True)) + else: + self.assertEqual(self.previous.read_text(), 'previous bytes') + + def test_latest_release_selection_installs_the_resolved_tag(self): + self.env.pop('STACK_VERSION') + result = self.run_installer() + self.assertEqual(result[0], 0, result) + self.assertIn(TAG, result[1]) + + def test_prerelease_cannot_become_latest_stable_install(self): + self.env.pop('STACK_VERSION') + self.latest['prerelease'] = True + self.assert_preserved(self.run_installer()) + + def test_reinstall_of_intact_release_succeeds(self): + self.assertEqual(self.run_installer()[0], 0) + result = self.run_installer() + self.assertEqual(result[0], 0, result) + + def test_tampered_existing_install_cannot_claim_verified_success(self): + self.assertEqual(self.run_installer()[0], 0) + destination = self.prefix / 'share' / CONFIG['repo'] / TAG + if CONFIG['kind'] == 'wheel': + target = self.previous.resolve() + elif CONFIG['kind'] == 'source': + target = destination / '.claude-plugin/plugin.json' + else: + target = destination / self.names[0] + target.write_text('tampered installed bytes') + result = self.run_installer() + self.assertNotEqual(result[0], 0, result) + self.assertNotIn('Installed verified', result[1]) + self.assertEqual(target.read_text(), 'tampered installed bytes') + + def test_partial_source_copy_failure_cleans_up_and_allows_retry(self): + if CONFIG['kind'] != 'source': + self.skipTest('source-copy failure applies to the source distribution') + destination = self.prefix / 'share' / CONFIG['repo'] / TAG + def partial_copy(source, target, **kwargs): + target.mkdir() + (target / 'partial').write_text('incomplete copy') + raise OSError('synthetic copy failure') + with patch('shutil.copytree', side_effect=partial_copy): + failed = self.run_installer() + self.assert_preserved(failed) + self.assertFalse(destination.exists(), 'partial final release survived failure') + releases = destination.parent + self.assertFalse((releases / '.install-lock').exists()) + self.assertFalse(list(releases.glob('.candidate-*'))) + retried = self.run_installer() + self.assertEqual(retried[0], 0, retried) + self.assertEqual(json.loads((destination / '.release.json').read_text())['tag'], TAG) + + def test_corrupt_checksum_preserves_previous(self): + path = self.release / self.names[0] + path.write_bytes(path.read_bytes() + b'corrupt') + self.assert_preserved(self.run_installer()) + + def test_missing_asset_preserves_previous(self): + (self.release / self.names[0]).unlink() + self.assert_preserved(self.run_installer()) + + def test_unsupported_platform_preserves_previous(self): + self.assert_preserved(self.run_installer(system='UnsupportedOS')) + + def test_missing_checksum_entry_preserves_previous(self): + (self.release / CONFIG['checksum']).write_text('') + self.assert_preserved(self.run_installer()) + + +if __name__ == '__main__': + unittest.main(verbosity=2)