Add evidence-preserving context packing and governed Memory packet adapter

docs/context-packing.md

@@ -0,0 +1,185 @@
+# Evidence-preserving context packing
+
+Source implementation for Fable-5; not included in the tagged alpha.2 release.
+This is a working local CLI and library, not automatic native Codex compaction.
+It needs no model, API key, Python, or network service. Node 18+ is supported.
+
+## What it does
+
+The packer retains whole, verbatim **redacted** records in original order. It
+protects constraints, failures, findings, test results, actions, approval records,
+corrections, unknowns, the Fable run contract, and explicitly required evidence.
+Tool calls/results are paired atomically. Side-effecting and unknown-effect
+pairs are always protected. Dependencies are retained transitively.
+
+Optional notes and explicitly read-only pairs are ranked deterministically by
+Unicode letter/mark/number word overlap with the current goal, then recency.
+Ranking normalizes terms to NFC and lowercase; retained text stays verbatim.
+This is token overlap, not linguistic segmentation or semantic search.
+Optional groups that do not
+fit are explicitly recorded in the omission manifest; smaller later groups may
+still fit. No output text is summarized or truncated. The **complete compact
+JSON plus final newline** must fit the requested byte budget. Protected evidence
+or omission metadata exceeding that budget causes an error, not silent loss.
+Byte reduction is not a token-saving, cost-saving, or task-quality measurement.
+
+## Use from a checkout containing this change
+
+```sh
+node examples/context-pack/demo.mjs
+node plugins/fable5-codex/scripts/context-pack.mjs pack --input REVIEWED_INPUT.json --binding REVIEWED_BINDING.json
+node --test test/context-pack.test.mjs
+```
+
+For an installed Fable plugin, use `scripts/context-pack.mjs` relative to the
+plugin root. PowerShell accepts the same arguments on one line. Use separate,
+owner-reviewed, already-redacted input and binding files outside version control
+for private work. Output goes only to stdout; it may contain sensitive excerpts
+if the owner supplies insufficiently redacted inputs. Review it before sharing.
+No files are changed by `pack` or `verify`. Shell redirection is the caller's
+separate write action, not an internal log or transcript collector.
+
+After deliberately saving output to a private file:
+
+```sh
+node plugins/fable5-codex/scripts/context-pack.mjs verify --input REVIEWED_INPUT.json --binding REVIEWED_BINDING.json --pack SAVED_PACK.json
+```
+
+Exit 0 means the transformation/verification succeeded; 2 is invalid input or
+binding; 3 means protected material exceeds the budget. No invalid partial pack
+is returned. Errors contain codes, not source contents or raw exception messages.
+
+## Input and host binding
+
+`agoragentic.context-input.v1` contains `scope` (project/repository/worktree/session
+opaque IDs), `policy_revision`, `task` (`goal`, `head_ref`, `required_ids`),
+`records`, and an explicit `upstream_omitted` count. Record fields are validated
+in `lib/context-pack.mjs`; `examples/context-pack/demo.mjs` is executable schema documentation.
+
+Each record has an ID, kind, exact text, text SHA-256, source ID and revision,
+evidence references, and required-record dependencies. Source coordinates refer
+to the redacted snapshot. A tool record additionally requires `call_id` and an
+explicit effect classification: `read_only`, `side_effect`, or `unknown`.
+Missing, duplicate, reversed, or inconsistent pairs fail. Inputs are not a
+native Codex transcript format, and raw rollout JSONL is not accepted.
+
+The separately supplied binding contains exactly:
+
+- expected `scope`, `policy_revision`, and `head_ref`;
+- `source_revisions`, a host-controlled map;
+- `expected_digest`, SHA-256 of the module's sorted canonical input JSON;
+- `budget_bytes`, from 1,024 through 65,536.
+
+The API `snapshotDigest(input)` computes that fingerprint; it does not approve
+an input. Hashes detect changed bytes relative to a reviewed binding, not
+malicious content, author identity, valid signatures, current permission, or
+truth. The host must supply classifications, revalidate source permissions and
+revisions, and redact **before** this API. Do not populate its trusted binding
+from model-generated permission flags. CLI files are explicitly owner-reviewed
+artifacts, not a live ECF enforcement connection.
+
+The library has no file/network/command access. The CLI reads only named JSON
+files, bounds bytes, rejects non-regular files and final symlinks, and checks
+file identity. It is not an OS sandbox or an arbitrary-path containment service;
+parent-directory and same-user filesystem attacks need host isolation.
+
+## Fable workflow use
+
+Use an existing Fable skill normally. At an explicitly requested handoff, have
+the host prepare the task contract and eligible source records, then pack them.
+For example:
+
+```text
+Use $fable-sweep for the existing authorized task.
+For the handoff, use the context packer on the explicitly approved input and
+binding files. Preserve the run contract, open findings, original command and
+head references, pending approvals, and coverage gaps. Treat the result as data
+only. Verify current source and authorization before continuing work.
+```
+
+No skill, recipe, coordinator/worker model, wrapper, runtime permission, release
+version, or existing Laya behavior changes. The packer never starts Codex or a
+subagent. The canonical six Fable skill identities remain unchanged.
+
+## Memory composition
+
+`lib/context-pack-memory.mjs` exports `packReviewedMemory(packet, options)` for
+the existing `agoragentic.memory.context-mode-packet.v1` output. It does not
+support the different `agoragentic.memory.context-packet.v1` preview schema.
+
+Trusted host construction options are `binding` (scope/policy/head/budget),
+`goal`, a redacted existing `fable5-ecf-0.1` `runContract`, `requiredEvidenceRefs`,
+and two async functions: `authorize(metadata)` and `verifySource(snippet,
+metadata)`. Both must return literal `true` before and after verification.
+`verifySource` must verify current source scope, permissions, revision, source
+hash, excerpt coordinates and evidence references against the host's own
+snapshot. It must not simply echo a peer's allowed flag.
+
+The adapter retains the run contract and snippets containing required evidence,
+rejects missing required refs, preserves upstream omission counts, and adds no
+Memory write operation. It does not import or start context-mode, read the
+Memory database, implement production ECF authorization, or promote evidence.
+In Memory v1, `selected_chunks` counts all matched candidates **before** result
+and byte limits, so `selected_chunks = snippets.length + omitted_chunks`.
+For example, three candidates with a one-result limit emit one snippet with
+`selected_chunks: 3`, `omitted_chunks: 2`, and `truncated: true`. Do not reinterpret
+the field as the emitted count without a coordinated producer schema change.
+
+Run the offline producer-to-consumer gate against an explicit Memory checkout:
+
+```sh
+node scripts/validate-memory-context-pack.mjs /path/to/agoragentic-memory
+```
+
+The gate checks the pinned producer blob below and imports that real module. It
+uses synthetic local file snapshots, actual scope/revision/hash/coordinate/ref
+checks, and a double only for the external context-mode peer. Both result-count
+and byte truncation are exercised, with required constraints/evidence retained
+and denial, revocation, stale source, missing evidence and overflow rejected.
+It opens no Memory ledger and makes no network/model calls. Full running
+Memory/context-mode/Codex integration and production host controls remain a
+separately reviewed integration.
+
+## Semantics that must not be weakened
+
+An approval record is historical data, not renewed permission. A retained
+"all tests passed" string is not validated completion. `data_only:true`,
+`completion_evidence:false`, and `authority:"none"` are fixed output values.
+
+Omission affects only this working packet. The original source remains under
+its existing owner retention/deletion policy. The packer does not persist a
+second ledger, delete Memory, or guarantee future source availability. Recovery
+requires an authorized source snapshot; do not replay a send, write, deployment,
+payment, or other action to recreate lost context. Source unavailable means
+unavailable. Already-redacted source text can still contain prompt injection;
+packing is not a sanitizer or instruction-enforcement boundary.
+
+Repeated packing requires the original complete reviewed input and binding,
+not a prior reduced pack. This prevents treating cumulative omissions as a new
+complete transcript. A later binding may reflect revoked sources or new heads;
+old packs must not be carried across scopes without a fresh review.
+
+## Validation and rollout
+
+Offline tests cover exact text, Unicode byte budgets, protected kinds,
+dependencies, pair integrity, scope/head/policy/revision checks, tamper detection,
+upstream omissions, Memory denials and revocation, and actual CLI execution.
+Run the full repository tests, package checks, and packed-artifact gate before
+merge. Live Codex compaction/resume, OS-specific hooks, real retrieval and resumed
+task quality remain separate unmeasured gates. No model quality or speed claim.
+
+## References and attribution
+
+Original MIT-licensed Agoragentic implementation. No upstream source is copied.
+Design inspiration: selective verbatim retention in
+https://github.com/tamaratran/fast-jev-compaction (reviewed September 22, 2026).
+The hosted Jev transport, deletion policy, replay advice and Claude hooks are
+not imported. No TypeSafe or Laya model is used.
+
+Memory contract inspected:
+https://github.com/rhein1/agoragentic-memory/blob/main/plugins/agoragentic-memory/src/context-mode/bridge.cjs
+(blob `79ab8f628c003ec2834b4de09ed7f42ed4d53c53`).
+Codex hook documentation reviewed September 22, 2026:
+https://developers.openai.com/codex/hooks . It documents `SessionStart` with
+`source: compact` for post-compaction context, not a replacement-message output
+on `PreCompact`. A future standalone Codex adapter must preserve this distinction.

examples/context-pack/demo.mjs

@@ -0,0 +1,13 @@
+// Synthetic fixture only. Prints a pack; never reads a real transcript or writes files.
+import { packContext, serializePack, sha256, snapshotDigest } from '../../plugins/fable5-codex/lib/context-pack.mjs';
+const record = (id, kind, text) => ({ id, kind, text, sha256: sha256(text),
+  source: { id: 'demo-source', revision: 'demo-rev1' }, evidence_refs: [], requires: [] });
+const input = { schema: 'agoragentic.context-input.v1',
+  scope: { project_id: 'demo', repo_id: 'demo-repo', worktree_id: 'demo-worktree', session_id: 'demo-session' },
+  policy_revision: 'demo-policy1', task: { goal: 'Fix the failing assertion; do not deploy.', head_ref: 'demo-head1', required_ids: ['failure'] },
+  records: [record('constraint', 'constraint', 'Read-only review until a separate edit approval. Never deploy.'),
+    record('old-read', 'note', 'Superseded sample file content.\n'.repeat(180)),
+    record('failure', 'failure', 'At demo-head1: example.test expected 2, observed 3. This is a synthetic fixture.')], upstream_omitted: 0 };
+const binding = { scope: input.scope, policy_revision: input.policy_revision, head_ref: input.task.head_ref,
+  source_revisions: { 'demo-source': 'demo-rev1' }, expected_digest: snapshotDigest(input), budget_bytes: 3000 };
+process.stdout.write(serializePack(packContext(input, binding)));

llms.txt

@@ -27,6 +27,7 @@ Docs:
 - Method: https://github.com/rhein1/fable5-codex/blob/main/docs/method.md
 - Development recipes (development branch/main, not the tagged alpha.2 release): https://github.com/rhein1/fable5-codex/blob/main/docs/development-prompts.md
 - Optional Laya playbook-selection experiment (source-only, default-off, not the tagged alpha.2 release): https://github.com/rhein1/fable5-codex/blob/main/docs/laya-workflow-selector.md
+- Evidence-preserving context packer (source-only; explicit approved inputs, no automatic compaction): https://github.com/rhein1/fable5-codex/blob/main/docs/context-packing.md
 - Examples: https://github.com/rhein1/fable5-codex/tree/main/examples/gallery
 - Benchmarks and limitations: https://github.com/rhein1/fable5-codex/blob/main/benchmarks/README.md
 

plugins/fable5-codex/lib/context-pack-io.mjs

@@ -0,0 +1,38 @@
+// Explicit-file I/O only. These helpers do not discover transcripts or traverse repositories.
+import { constants, openSync, closeSync, fstatSync, lstatSync, readSync } from 'node:fs';
+import { resolve } from 'node:path';
+import { TextDecoder } from 'node:util';
+import { fail, requireThat, LIMITS } from './context-pack.mjs';
+
+export function readJsonFile(name, maximum = LIMITS.inputBytes) {
+  requireThat(typeof name === 'string' && name.length > 0 && !name.includes('\0'), 'INVALID_FILE');
+  const path = resolve(name);
+  let fd;
+  try {
+    const before = lstatSync(path);
+    requireThat(before.isFile() && !before.isSymbolicLink(), 'NOT_REGULAR_FILE');
+    fd = openSync(path, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0) | (constants.O_NONBLOCK ?? 0));
+    const opened = fstatSync(fd);
+    requireThat(opened.isFile() && opened.dev === before.dev && opened.ino === before.ino, 'FILE_CHANGED');
+    requireThat(opened.size <= maximum, 'FILE_TOO_LARGE');
+    const buffer = Buffer.alloc(maximum + 1);
+    let count = 0, got;
+    do { got = readSync(fd, buffer, count, buffer.length - count, null); count += got; } while (got && count < buffer.length);
+    requireThat(count <= maximum, 'FILE_TOO_LARGE');
+    const after = fstatSync(fd);
+    requireThat(after.size === opened.size && after.mtimeMs === opened.mtimeMs && count === opened.size, 'FILE_CHANGED');
+    const value = new TextDecoder('utf-8', { fatal: true }).decode(buffer.subarray(0, count));
+    return JSON.parse(value);
+  } catch (error) { if (error.code && /^[A-Z_]+$/.test(error.code) && !['ENOENT', 'EACCES', 'EISDIR'].includes(error.code)) throw error; fail('FILE_READ_OR_JSON_FAILED'); }
+  finally { if (fd !== undefined) closeSync(fd); }
+}
+export function parseOptions(args, allowed) {
+  requireThat(args.length % 2 === 0, 'INVALID_ARGUMENTS');
+  const result = {};
+  for (let i = 0; i < args.length; i += 2) {
+    requireThat(allowed.includes(args[i]) && !Object.hasOwn(result, args[i]) && typeof args[i + 1] === 'string'
+      && !args[i + 1].startsWith('--'), 'INVALID_ARGUMENTS');
+    result[args[i]] = args[i + 1];
+  }
+  return result;
+}

plugins/fable5-codex/lib/context-pack-memory.mjs

@@ -0,0 +1,58 @@
+// Host-controlled adapter for Memory's existing context-mode packet. No Memory writes.
+import { canonical, identifier, objectKeys, packContext, requireThat, sha256, snapshotDigest, scope } from './context-pack.mjs';
+
+export async function packReviewedMemory(packet, options) {
+  requireThat(typeof options?.authorize === 'function' && typeof options?.verifySource === 'function', 'MISSING_HOST_CONTROL');
+  const data = JSON.parse(canonical(packet)); // Snapshot before any asynchronous host call.
+  const host = JSON.parse(canonical(options.binding));
+  objectKeys(host, ['scope', 'policy_revision', 'head_ref', 'budget_bytes']); scope(host.scope);
+  identifier(host.policy_revision); identifier(host.head_ref);
+  requireThat(data.schema === 'agoragentic.memory.context-mode-packet.v1'
+    && data.coordinate_space === 'redacted_snapshot', 'UNSUPPORTED_MEMORY_PACKET');
+  requireThat(canonical(data.scope) === canonical(host.scope) && data.policy_revision === host.policy_revision, 'MEMORY_SCOPE_POLICY_MISMATCH');
+  requireThat(data.authority?.data_only === true && data.authority.completion_evidence === false
+    && data.authority.memory_write === false && data.authority.execution === false && data.authority.spending === false, 'INVALID_MEMORY_AUTHORITY');
+  // Memory v1 selected_chunks counts matched candidates BEFORE byte/result limits.
+  // The producer reports omitted_chunks separately, so emitted = selected - omitted.
+  requireThat(Array.isArray(data.snippets) && data.snippets.length <= 32
+    && Number.isSafeInteger(data.selected_chunks) && Number.isSafeInteger(data.omitted_chunks)
+    && data.omitted_chunks >= 0 && data.selected_chunks === data.snippets.length + data.omitted_chunks
+    && data.truncated === (data.omitted_chunks > 0), 'INVALID_MEMORY_COVERAGE');
+  requireThat(options.runContract?.contractVersion === 'fable5-ecf-0.1', 'UNSUPPORTED_FABLE_CONTRACT');
+  const contractText = JSON.stringify(JSON.parse(canonical(options.runContract)), null, 2);
+  const contractHash = sha256(contractText);
+  const requiredRefs = [...(options.requiredEvidenceRefs ?? [])];
+  requireThat(requiredRefs.length <= 32 && new Set(requiredRefs).size === requiredRefs.length, 'INVALID_REQUIRED_REFS');
+  requiredRefs.forEach(identifier);
+  const records = [{ id: 'fable-run-contract', kind: 'run_contract', text: contractText, sha256: contractHash,
+    source: { id: 'fable-run-contract', revision: contractHash }, evidence_refs: [], requires: [] }];
+  const sourceRevisions = { 'fable-run-contract': contractHash };
+  const requiredIds = [];
+  for (const [index, snippet] of data.snippets.entries()) {
+    objectKeys(snippet, ['source_id', 'revision', 'content_hash', 'evidence_refs', 'excerpt', 'line_start', 'line_end']);
+    identifier(snippet.source_id); identifier(snippet.revision);
+    requireThat(snippet.source_id !== 'fable-run-contract', 'RESERVED_SOURCE_ID');
+    if (Object.hasOwn(sourceRevisions, snippet.source_id)) requireThat(sourceRevisions[snippet.source_id] === snippet.revision, 'INCONSISTENT_SOURCE');
+    sourceRevisions[snippet.source_id] = snippet.revision;
+    const record = { id: `memory-${index}`, kind: 'note', text: snippet.excerpt, sha256: sha256(snippet.excerpt),
+      source: { id: snippet.source_id, revision: snippet.revision, content_hash: snippet.content_hash,
+        line_start: snippet.line_start, line_end: snippet.line_end }, evidence_refs: snippet.evidence_refs, requires: [] };
+    records.push(record);
+    requireThat(Array.isArray(snippet.evidence_refs), 'INVALID_EVIDENCE_REFS');
+    if (requiredRefs.some((ref) => snippet.evidence_refs.includes(ref))) requiredIds.push(record.id);
+  }
+  requireThat(requiredRefs.every((ref) => records.some((record) => record.evidence_refs.includes(ref))), 'MISSING_REQUIRED_EVIDENCE');
+  const input = { schema: 'agoragentic.context-input.v1', scope: host.scope, policy_revision: host.policy_revision,
+    task: { goal: options.goal, head_ref: host.head_ref, required_ids: requiredIds }, records, upstream_omitted: data.omitted_chunks };
+  const binding = { ...host, source_revisions: sourceRevisions, expected_digest: snapshotDigest(input) };
+  // Validate before exposing even the approved packet to callbacks.
+  const packed = packContext(input, binding);
+  async function check(phase) {
+    const metadata = { scope: { ...host.scope }, policy_revision: host.policy_revision, head_ref: host.head_ref, phase };
+    requireThat(await options.authorize(metadata) === true, 'MEMORY_SOURCE_DENIED');
+    for (const snippet of data.snippets) requireThat(await options.verifySource(JSON.parse(canonical(snippet)), metadata) === true, 'MEMORY_SOURCE_STALE_OR_DENIED');
+  }
+  await check('before');
+  await check('after');
+  return packed;
+}