diff --git a/README.md b/README.md index d44e264..f037ac3 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,10 @@ building, and publishing generated hub artifacts. ## Usage +For optional concise-response hooks and experimental input compaction, see +[token-efficient mode](docs/token-efficiency.md), including measured savings, +research, supported hosts, and rollback instructions. + Use the [GitHub workflows](#github-actions) or [GitLab template](#gitlab-ci) to keep toolchain setup in this repository. For custom GitHub jobs, see [direct action usage](#direct-action-usage). diff --git a/docs/token-efficiency.md b/docs/token-efficiency.md new file mode 100644 index 0000000..8026d7c --- /dev/null +++ b/docs/token-efficiency.md @@ -0,0 +1,172 @@ +# Token-efficient mode + +PIG supports two independent, opt-in settings. Both default to `false`: + +```yaml +token_efficiency: + concise_output: true + compact_instructions: false +``` + +`concise_output` installs a Claude hook in the canonical `pig` plugin. At +`SessionStart` and `SubagentStart`, it asks for concise prose while preserving +requirements, necessary reasoning, uncertainty, exact literals, and requested +detail. It explicitly forbids skipping work or checks to save tokens. It does +not change model selection, reasoning effort, tool permissions, or task scope. + +`compact_instructions` is experimental input rewriting. It uses the same +deterministic, local compactor in two places: + +- At build time, generated skill entrypoints in authored plugins, including + converted agent/command skills, across the configured target hosts. MCP + distribution also compacts skill entrypoints, agent projections, and command + prompt bodies, with hashes and sizes calculated from the distributed bytes. +- In Claude, a `PreToolUse` hook for `Agent` and legacy `Task` rewrites only the + `prompt` argument. It retains all other arguments and leaves permission + decisions to the host. + +Authored files are never rewritten. References, scripts, routing frontmatter, +and external plugins retain their authored bytes. Native rules, agents, and +commands outside generated skill entrypoints keep their normal rendering. +There is no proxy, runtime model call, or tool-output compression. Typed user +prompts cannot be replaced by `UserPromptSubmit`; that hook can add context, +which would increase input tokens. Codex, Cursor, Gemini, and MCP get the +generated content option but no token-efficiency runtime hook in this version. + +## Install, inspect, and disable + +Set the flags in `hub.yaml`, then run `pig verify --hub .` and publish through +the hub's normal PR/release workflow. Update the installed `pig` plugin and any +plugins whose generated skills changed, then start a fresh session. Managed +files belong to the toolchain; do not add hook files to the hub's source assets. + +The Claude hook requires `python3` (3.9+) on the application's PATH and a host +version supporting `PreToolUse.updatedInput` for input rewriting. Local command +execution was tested on macOS and Linux; Windows/native desktop installation +has not been tested. It emits no context for unchanged or unknown inputs. +Malformed, oversized (over 1 MiB), or excessively nested JSON is skipped with a +generic stderr message that does not include prompt content. + +Preview the exact input transformation without writing files: + +```bash +pig compact-preview assets/skills/example/SKILL.md +``` + +The preview reports UTF-8 bytes, **not tokens**. To measure the actual corpus +with an explicitly named tokenizer, from a toolchain checkout run: + +```bash +uv run --with tiktoken python scripts/measure_token_efficiency.py /path/to/hub --diff +``` + +Set either flag to `false` and rebuild/update to remove its behavior. Generated +skills revert to the source text when `compact_instructions` is disabled. +`PIG_TOKEN_EFFICIENT_MODE=off` in the Claude process environment disables future +runtime hook output. It cannot restore generated files or remove context +already injected into a running conversation; restart the session after +updating the plugin or environment. + +## Why this design + +[Caveman's hook configuration](https://github.com/JuliusBrussee/caveman/blob/d262ab3882952a59312ff3537cd86d894523c5ab/.claude-plugin/plugin.json) +uses session/subagent activation to inject concise-writing guidance. Its +separate prompt tracking hook does not delete words from the user's message. +PIG follows the context-injection approach for output, using a short instruction +instead of loading an entire style skill on every activation. + +The [CAVEWOMAN study](https://arxiv.org/abs/2606.24083) found that its evaluated +input word-deletion methods could hurt accuracy and increase response length; +output constraints were more promising on its tested tasks. This is evidence +against treating grammatical words as universally disposable, not a guarantee +that any concise style is harmless. [LLMLingua-2](https://arxiv.org/abs/2403.12968) +uses a trained context-aware compressor and task evaluations. Its results do +not establish that generic stopword removal preserves instruction semantics. + +[Anthropic's context engineering guidance](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) +favors high-signal context and progressive retrieval. Keep skill routing +descriptions precise and move optional detail into referenced resources before +trying aggressive rewriting. Stable, deterministic build output also preserves +prefix reuse better than per-request paraphrasing; the importance of exact +prefixes follows from the [prompt caching contract](https://platform.claude.com/docs/en/build-with-claude/prompt-caching). + +The compactor therefore has only seven reviewed phrase substitutions: + +| Original | Replacement | +| --- | --- | +| in order to | to | +| due to the fact that | because | +| at this point in time | now | +| in the event that | if | +| for the purpose of | for | +| with the exception of | except | +| on a daily basis | daily | + +It retains whitespace, frontmatter, fenced/indented code, and headings, and +skips paragraphs containing possible literals such as quotes, inline code, +links, paths, tables, HTML, or JSON. Multiline quotes cause the entire input to +be retained. This intentionally leaves many compression opportunities unused. +Even these substitutions are not provably lossless; exact wording can be part +of a task without being marked as a literal. Review diffs and keep input +rewriting disabled unless a representative evaluation supports it. + +Hook event and replacement behavior follows the +[Claude hooks reference](https://code.claude.com/docs/en/hooks). `updatedInput` +replaces the input object, so the hook copies every unchanged argument and +does not emit a permission override. + +## Initial evidence (2026-10-16) + +The actual Promptless hub at `fdfecb7842e423176d01956885f1d7c56ffd4ce0` has 69 +authored skill entrypoints. With `o200k_base`, the compactor produced **116,660 +tokens before and after: zero savings**. None contained the seven phrases. +This counts each source entrypoint once, not an entire session or every plugin +duplicate. It excludes references, dynamically loaded context, and provider +billing. The concise instruction itself costs 65 tokens under this tokenizer +per activation; Claude's tokenizer and host wrapping differ. + +A tool-disabled Claude evaluation used the 12 synthetic cases in +[`cases.json`](../tests/fixtures/token-efficiency/cases.json): approvals, +negation, exceptions, conditions, quantities, units, literals, code, +prerequisites, scope, uncertainty, and cause. Each of four arms answered the +same 12 independent cases in one request, with one sample per arm on each of +macOS and Linux. Both runs passed every case. The table shows the Linux rerun +with the committed evaluator; the reported model was `claude-opus-5-5`. + +| Arm | Exact matches | Input tokens including cache creation/read | Output tokens | +| --- | --- | --- | --- | +| Baseline | 12/12 | 1,214 | 136 | +| Input compaction | 12/12 | 1,187 | 136 | +| Concise output | 12/12 | 1,329 | 136 | +| Both | 12/12 | 1,302 | 136 | + +The fixed answer format leaves no response prose to shorten, so the concise +instruction adds overhead here. These results support preserving the tested +constraints; they do **not** demonstrate whole-session savings or unchanged +coding performance. They also do not test a desktop host loading the plugin. + +A separate Linux Claude CLI smoke test loaded the generated plugin from a path +containing spaces. In a fresh session the model received the exact +`SessionStart` instruction; with `PIG_TOKEN_EFFICIENT_MODE=off`, it received no +such instruction. `SubagentStart` and `PreToolUse` are covered by contract and +subprocess tests, not a native end-to-end delegation test. + +To repeat the paid smoke test with authenticated Claude CLI, from a toolchain +checkout run: + +```bash +uv run python scripts/evaluate_token_efficiency.py --model claude-opus-5-5 +``` + +The runner disables tools, installed plugins/hooks through safe mode, slash commands, +settings, and MCP; uses a temporary working directory; and sends only synthetic +fixtures. The compactor and concise instruction are supplied directly to +isolate their effects. It caps each of four requests at $2, disables session +persistence, and prints scores plus provider usage without session identifiers. +Model availability, tokenization, and sampling can change the result. + +Before enabling input rewriting more broadly, compare representative coding +and skill-use tasks on supported models with repeated trials. Gate on task +completion, authorization/negation/literal preservation, tool validity, and +required verification, and measure total input plus output tokens, cache usage, +and latency. A smaller prompt alone is not an acceptance criterion. diff --git a/schemas/instruction-hub.schema.json b/schemas/instruction-hub.schema.json index 10dc86d..a463536 100644 --- a/schemas/instruction-hub.schema.json +++ b/schemas/instruction-hub.schema.json @@ -48,6 +48,15 @@ "properties": {"enabled": {"type": "boolean", "default": false}}, "default": {"enabled": false} }, + "token_efficiency": { + "type": "object", + "additionalProperties": false, + "properties": { + "concise_output": {"type": "boolean", "default": false}, + "compact_instructions": {"type": "boolean", "default": false} + }, + "default": {"concise_output": false, "compact_instructions": false} + }, "trace_ingestion": { "type": "object", "additionalProperties": false, diff --git a/scripts/evaluate_token_efficiency.py b/scripts/evaluate_token_efficiency.py new file mode 100644 index 0000000..e441101 --- /dev/null +++ b/scripts/evaluate_token_efficiency.py @@ -0,0 +1,101 @@ +"""Opt-in, paid model smoke test of synthetic constraints; never reads local sessions. + +Run: uv run python scripts/evaluate_token_efficiency.py --model MODEL_ID +Runs four requests, each containing the same 12 independent tasks. This is a +constraint smoke test, not a quality benchmark or native plugin acceptance test. +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import tempfile +from pathlib import Path + +from promptless_instruction_hub.token_efficiency import CONCISE_INSTRUCTION, compact_markdown + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model", required=True, help="explicit Claude model ID for reproducibility") + args = parser.parse_args() + fixture = Path(__file__).resolve().parents[1] / "tests/fixtures/token-efficiency/cases.json" + cases = json.loads(fixture.read_text()) + any_failures = False + with tempfile.TemporaryDirectory(prefix="pig-token-eval-") as directory: + for mode in ("baseline", "compact", "concise", "both"): + inputs = [ + { + "id": case["id"], + "prompt": compact_markdown(case["prompt"]) if mode in ("compact", "both") else case["prompt"], + } + for case in cases + ] + prompt = "Answer each independent task. Return only a JSON object mapping each id to its answer.\n" + prompt += json.dumps(inputs) + system = "Follow task instructions exactly." + if mode in ("concise", "both"): + system += "\n" + CONCISE_INSTRUCTION + result = subprocess.run( + [ + "claude", + "-p", + "--model", + args.model, + "--safe-mode", + "--tools", + "", + "--disable-slash-commands", + "--setting-sources", + "", + "--strict-mcp-config", + "--mcp-config", + '{"mcpServers":{}}', + "--system-prompt", + system, + "--output-format", + "json", + "--no-session-persistence", + "--max-budget-usd", + "2", + ], + input=prompt, + text=True, + capture_output=True, + cwd=directory, + timeout=180, + check=True, + ) + data = json.loads(result.stdout) + if isinstance(data, list): + data = next(event for event in reversed(data) if event.get("type") == "result") + if data.get("is_error"): + raise RuntimeError(f"{mode}: Claude returned an error result") + answer = data["result"].strip() + if answer.startswith("```"): + answer = "\n".join(answer.splitlines()[1:-1]) + answers = json.loads(answer) + failures = [case["id"] for case in cases if answers.get(case["id"]) != case["expected"]] + any_failures |= bool(failures) + print( + json.dumps( + { + "mode": mode, + "passed": len(cases) - len(failures), + "total": len(cases), + "failed_ids": failures, + "requested_model": args.model, + "models": sorted(data.get("modelUsage", {})), + "usage": data.get("usage"), + "cost_usd": data.get("total_cost_usd"), + } + ), + flush=True, + ) + if any_failures: + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/scripts/measure_token_efficiency.py b/scripts/measure_token_efficiency.py new file mode 100644 index 0000000..e75738d --- /dev/null +++ b/scripts/measure_token_efficiency.py @@ -0,0 +1,66 @@ +"""Measure local SKILL.md savings with an explicitly named tokenizer. + +Run: uv run --with tiktoken python scripts/measure_token_efficiency.py HUB +No model calls, source writes, or session/trace reads. +""" + +from __future__ import annotations + +import argparse +import difflib +import importlib +import json +from pathlib import Path + +from promptless_instruction_hub.token_efficiency import CONCISE_INSTRUCTION, compact_markdown + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("hub", type=Path) + parser.add_argument("--encoding", default="o200k_base") + parser.add_argument("--diff", action="store_true") + args = parser.parse_args() + try: + tiktoken = importlib.import_module("tiktoken") + except ModuleNotFoundError: + parser.error("install the measurement-only dependency with uv run --with tiktoken") + encoding = getattr(tiktoken, "get_encoding")(args.encoding) + files = sorted((args.hub / "assets/skills").glob("*/SKILL.md")) + if not files: + parser.error("no assets/skills/*/SKILL.md files found") + rows = [] + for path in files: + original = path.read_bytes().decode("utf-8") + compacted = compact_markdown(original) + rows.append( + { + "file": path.relative_to(args.hub).as_posix(), + "before": len(encoding.encode(original, disallowed_special=())), + "after": len(encoding.encode(compacted, disallowed_special=())), + "changed": original != compacted, + } + ) + if args.diff and original != compacted: + print( + "".join( + difflib.unified_diff(original.splitlines(True), compacted.splitlines(True), str(path), "compacted") + ) + ) + print( + json.dumps( + { + "encoding": args.encoding, + "scope": "authored skill entrypoints, counted once each; not whole-session usage or provider billing", + "before": sum(row["before"] for row in rows), + "after": sum(row["after"] for row in rows), + "concise_hook_context_tokens_per_activation": len(encoding.encode(CONCISE_INSTRUCTION)), + "files": rows, + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/src/promptless_instruction_hub/cli.py b/src/promptless_instruction_hub/cli.py index b7490e0..0977ac0 100644 --- a/src/promptless_instruction_hub/cli.py +++ b/src/promptless_instruction_hub/cli.py @@ -109,6 +109,11 @@ def _build_parser() -> argparse.ArgumentParser: "--allowed-host", action="append", help="additional allowed HTTP Host, optionally with :port or :*" ) + preview_parser = subcommands.add_parser( + "compact-preview", help="preview conservative prose compaction without writing files" + ) + preview_parser.add_argument("file", type=Path) + return parser @@ -117,6 +122,11 @@ def _add_hub_arg(parser: argparse.ArgumentParser) -> None: def _dispatch(args: argparse.Namespace) -> int: + if args.command == "compact-preview": + from promptless_instruction_hub.token_efficiency_preview import preview_file + + print(preview_file(args.file), end="") + return 0 if args.command == "serve-mcp": try: from promptless_instruction_hub.mcp_distribution.server import serve diff --git a/src/promptless_instruction_hub/mcp_distribution/compiler.py b/src/promptless_instruction_hub/mcp_distribution/compiler.py index 91c9c37..46260ae 100644 --- a/src/promptless_instruction_hub/mcp_distribution/compiler.py +++ b/src/promptless_instruction_hub/mcp_distribution/compiler.py @@ -26,6 +26,7 @@ from promptless_instruction_hub.models import LoadedAsset, ResolvedExternalPluginDefinition, ResolvedHubPluginDefinition from promptless_instruction_hub.release.hashing import stable_hash from promptless_instruction_hub.validate.hub import ValidationResult +from promptless_instruction_hub.token_efficiency import compact_markdown BUNDLE_PATH = Path("dist/mcp") MANIFEST_NAME = "bundle.json" @@ -85,7 +86,12 @@ def render_mcp_bundle(output_root: Path, validation: ValidationResult[ResolvedHu ) for asset in plugin.assets: memberships.setdefault(asset.ref, []).append(definition.id) - assets = [_compile_asset(root, asset, sorted(memberships[asset.ref])) for asset in validation.stable_assets] + assets = [ + _compile_asset( + root, asset, sorted(memberships[asset.ref]), compact=validation.config.token_efficiency.compact_instructions + ) + for asset in validation.stable_assets + ] project_skill_tools(assets) files = [file for asset in assets for file in asset.files] if len(files) > MAX_BUNDLE_FILES or sum(file.size for file in files) > MAX_BUNDLE_BYTES: @@ -112,7 +118,7 @@ def render_mcp_bundle(output_root: Path, validation: ValidationResult[ResolvedHu raise InstructionHubError("MCP bundle manifest exceeds 16 MiB") -def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str]) -> BundleAsset: +def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str], *, compact: bool = False) -> BundleAsset: relative_root = Path("assets") / asset.type / asset.id sources = sorted(asset.path.rglob("*")) if asset.path.is_dir() else [asset.path] files: list[BundleFile] = [] @@ -124,7 +130,10 @@ def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str]) -> Bundle if not source.is_file(): raise InstructionHubError(f"{asset.ref}: only regular files can be distributed") relative = source.relative_to(asset.path) if asset.path.is_dir() else Path(source.name) - files.append(_write_file(root, relative_root / relative, source.read_bytes())) + data = source.read_bytes() + if compact and asset.type == "skill" and relative.parent == Path(".") and relative.name.lower() == "skill.md": + data = compact_markdown(data.decode("utf-8")).encode("utf-8") + files.append(_write_file(root, relative_root / relative, data)) result = BundleAsset( ref=asset.ref, id=asset.id, @@ -150,7 +159,7 @@ def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str]) -> Bundle original = entrypoints[0] content = (root / original.path).read_bytes() frontmatter = read_frontmatter(content) - # Normalize filename casing without changing the authored bytes. + # Normalize filename casing without further changing the content. entrypoint = relative_root / "SKILL.md" if original.path != entrypoint.as_posix(): (root / original.path).unlink() @@ -160,7 +169,8 @@ def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str]) -> Bundle frontmatter=frontmatter, entrypoint=entrypoint.as_posix(), files=sorted(file.path for file in files) ) elif asset.type == "agent": - content = render_agent_skill(read_agent_skill(asset)).encode("utf-8") + procedure = render_agent_skill(read_agent_skill(asset)) + content = (compact_markdown(procedure) if compact else procedure).encode("utf-8") entrypoint = Path("projections/agent") / asset.id / "SKILL.md" result.files.append(_write_file(root, entrypoint, content)) result.skill = BundleSkill( @@ -169,7 +179,11 @@ def _compile_asset(root: Path, asset: LoadedAsset, plugins: list[str]) -> Bundle elif asset.type == "command": # The Codex conversion validates the host-independent, explicit-invocation subset. command = read_command(asset, "codex") - result.prompt = BundlePrompt(name=f"command:{asset.id}", description=command.description, body=command.body) + result.prompt = BundlePrompt( + name=f"command:{asset.id}", + description=command.description, + body=compact_markdown(command.body) if compact else command.body, + ) if result.skill is not None: skill_files = [file for file in result.files if file.path in result.skill.files] if len(skill_files) > MAX_SKILL_FILES or sum(file.size for file in skill_files) > MAX_SKILL_BYTES: diff --git a/src/promptless_instruction_hub/models.py b/src/promptless_instruction_hub/models.py index 34be495..a37de94 100644 --- a/src/promptless_instruction_hub/models.py +++ b/src/promptless_instruction_hub/models.py @@ -105,6 +105,15 @@ class McpDistributionConfig(BaseModel): enabled: bool = Field(default=False, strict=True) +class TokenEfficiencyConfig(BaseModel): + """Independent, opt-in output guidance and conservative input compaction.""" + + model_config = ConfigDict(extra="forbid") + + concise_output: bool = Field(default=False, strict=True) + compact_instructions: bool = Field(default=False, strict=True) + + class HubConfig(BaseModel): """Root `hub.yaml` configuration.""" @@ -117,6 +126,7 @@ class HubConfig(BaseModel): targets: list[Harness] = Field(default_factory=lambda: list(SUPPORTED_HARNESSES), min_length=1) trace_ingestion: TraceIngestionConfig = Field(default_factory=TraceIngestionConfig) mcp: McpDistributionConfig = Field(default_factory=McpDistributionConfig) + token_efficiency: TokenEfficiencyConfig = Field(default_factory=TokenEfficiencyConfig) @field_validator("version") @classmethod diff --git a/src/promptless_instruction_hub/render/plugins.py b/src/promptless_instruction_hub/render/plugins.py index 0cf215b..d8d1d82 100644 --- a/src/promptless_instruction_hub/render/plugins.py +++ b/src/promptless_instruction_hub/render/plugins.py @@ -22,6 +22,7 @@ from promptless_instruction_hub.render.assets import render_assets_for_target from promptless_instruction_hub.render.common import RenderedAssets from promptless_instruction_hub.render.mcp import collect_mcp_servers, write_mcp_config +from promptless_instruction_hub.render.token_efficiency import render_token_efficiency def render_target_plugins( @@ -50,6 +51,7 @@ def render_target_plugins( render_managed_skills(target_root, target, config, stable_plugin.definition) ) managed_runtimes.extend(render_managed_runtimes(target_root, target, config, stable_plugin.definition)) + render_token_efficiency(target_root, target, config, stable_plugin.definition) _write_manifest(target_root, target, config, stable_plugin.definition, rendered, mcp_servers) if "codex" in config.targets: codex.write_marketplace(output_root, config, plugins) diff --git a/src/promptless_instruction_hub/render/token_efficiency.py b/src/promptless_instruction_hub/render/token_efficiency.py new file mode 100644 index 0000000..d2039f9 --- /dev/null +++ b/src/promptless_instruction_hub/render/token_efficiency.py @@ -0,0 +1,52 @@ +"""Opt-in generated skill compaction and Claude hooks in the canonical PIG plugin.""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +from promptless_instruction_hub import token_efficiency +from promptless_instruction_hub.errors import InstructionHubError +from promptless_instruction_hub.fs import JsonValue, read_json_mapping, write_json +from promptless_instruction_hub.models import PIG_PLUGIN_ID, Harness, HubConfig, PluginDefinition + + +def render_token_efficiency(target_root: Path, target: Harness, config: HubConfig, plugin: PluginDefinition) -> None: + mode = config.token_efficiency + if mode.compact_instructions: + # Only skill entrypoints. Resources, scripts, native rules and metadata + # remain authored bytes, and external plugin directories never reach here. + for directory in sorted((target_root / "skills").glob("*")): + if directory.is_dir(): + for path in sorted(directory.iterdir()): + if path.is_file() and path.name.lower() == "skill.md": + content = path.read_bytes().decode("utf-8") + path.write_bytes(token_efficiency.compact_markdown(content).encode("utf-8")) + if target != "claude" or plugin.id != PIG_PLUGIN_ID or not (mode.concise_output or mode.compact_instructions): + return + hook_root = target_root / "hooks" + hook_root.mkdir(parents=True, exist_ok=True) + runner = hook_root / "pig-token-efficiency.py" + if runner.exists(): + raise InstructionHubError(f"token efficiency hook conflicts with authored file: {runner}") + shutil.copy2(token_efficiency.__file__, runner) + path = hook_root / "hooks.json" + content = read_json_mapping(path) if path.exists() else {} + events = content.setdefault("hooks", {}) + if not isinstance(events, dict): + raise InstructionHubError(f"{path}: hooks must be an object") + command = 'python3 "${CLAUDE_PLUGIN_ROOT}/hooks/pig-token-efficiency.py"' + bindings = [] + if mode.concise_output: + bindings.extend([("SessionStart", None, "--concise"), ("SubagentStart", None, "--concise")]) + if mode.compact_instructions: + bindings.append(("PreToolUse", "Agent|Task", "--compact")) + for event, matcher, flag in bindings: + handlers = events.setdefault(event, []) + if not isinstance(handlers, list): + raise InstructionHubError(f"{path}: {event} must be an array") + entry: dict[str, JsonValue] = {"hooks": [{"type": "command", "command": f"{command} {flag}", "timeout": 5}]} + if matcher: + entry["matcher"] = matcher + handlers.append(entry) + write_json(path, content) diff --git a/src/promptless_instruction_hub/token_efficiency.py b/src/promptless_instruction_hub/token_efficiency.py new file mode 100644 index 0000000..7d063ca --- /dev/null +++ b/src/promptless_instruction_hub/token_efficiency.py @@ -0,0 +1,151 @@ +"""Conservative local prose compaction and a standalone Claude hook (Python 3.9+). + +This module is copied into plugins: keep it standard-library-only. Rewriting +natural language is not provably lossless; callers must explicitly opt in. +""" + +from __future__ import annotations + +import json +import os +import re +import sys +from typing import Any + +# Deliberately omit stopwords, modal verbs, negations, quantities and qualifiers. +# Keep this small enough for reviewers to audit every transformation. +_REDUCTIONS = { + "in order to": "to", + "due to the fact that": "because", + "at this point in time": "now", + "in the event that": "if", + "for the purpose of": "for", + "with the exception of": "except", + "on a daily basis": "daily", +} +# Fold ASCII phrase letters only, retaining Unicode-aware outer word boundaries. +_PHRASES = re.compile(r"\b(?ai:" + "|".join(re.escape(phrase) for phrase in _REDUCTIONS) + r")\b") +# A paragraph containing possible literals is left entirely intact. This also +# protects multiline quoted values, links, tables, HTML, paths and inline code. +_LITERAL_MARKERS = re.compile(r"[`\[\]{}<>|\\/\"'“”‘’]") +_FENCE = re.compile(r"^ {0,3}(`{3,}|~{3,})") +MAX_INPUT_BYTES = 1024 * 1024 + +CONCISE_INSTRUCTION = ( + "Use concise prose: answer first; omit greetings, filler and repeated summaries. " + "Preserve all requirements, reasoning needed to assess the answer, uncertainty, " + "negations, conditions, numbers, units, and exact code, commands, paths and errors. " + "Honor requested detail and formats. Never skip work, checks or necessary explanations to save tokens." +) + + +def _replace(match: re.Match[str]) -> str: + phrase = match.group() + replacement = _REDUCTIONS[phrase.lower()] + if phrase.isupper(): + return replacement.upper() + if phrase[0].isupper(): + return replacement[0].upper() + replacement[1:] + return replacement + + +def compact_markdown(text: str) -> str: + """Reduce reviewed phrases in plain paragraphs, preserving Markdown structure. + + Frontmatter, fenced/indented code and paragraphs containing possible literal + syntax are opaque. Whitespace is never normalized. No model or network runs. + """ + for literal in re.finditer(r""""(?:\\.|[^"\\])*"|(? None: + content = "".join(paragraph) + if not _LITERAL_MARKERS.search(content): + content = _PHRASES.sub(_replace, content) + output.append(content) + paragraph.clear() + + for index, line in enumerate(lines): + if frontmatter: + output.append(line) + if index > 0 and line.strip() in {"---", "..."}: + frontmatter = False + continue + fence = _FENCE.match(line) + if fence_char: + output.append(line) + if ( + fence + and fence[1][0] == fence_char + and len(fence[1]) >= fence_length + and not line[fence.end() :].strip() + ): + fence_char = "" + continue + if fence: + flush() + fence_char, fence_length = fence[1][0], len(fence[1]) + output.append(line) + elif not line.strip() or line.startswith((" ", "\t", "#")): + flush() + output.append(line) + else: + paragraph.append(line) + flush() + return "".join(output) + + +def hook_response(payload: Any, *, concise: bool, compact: bool) -> dict[str, object] | None: + """Use only host fields that change context or Agent/Task input, never permissions.""" + if not isinstance(payload, dict): + return None + event = payload.get("hook_event_name") + if concise and event in ("SessionStart", "SubagentStart"): + return {"hookSpecificOutput": {"hookEventName": event, "additionalContext": CONCISE_INSTRUCTION}} + if not compact or event != "PreToolUse" or payload.get("tool_name") not in ("Agent", "Task"): + return None + tool_input = payload.get("tool_input") + if not isinstance(tool_input, dict) or not isinstance(tool_input.get("prompt"), str): + return None + prompt = tool_input["prompt"] + compressed = compact_markdown(prompt) + if compressed == prompt: + return None + return { + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "updatedInput": {**tool_input, "prompt": compressed}, + } + } + + +def main() -> int: + """Fail open without echoing prompt data; a disabled hook emits no context.""" + if os.environ.get("PIG_TOKEN_EFFICIENT_MODE", "").lower() == "off": + return 0 + try: + raw = sys.stdin.buffer.read(MAX_INPUT_BYTES + 1) + if len(raw) > MAX_INPUT_BYTES: + print("PIG token efficiency: oversized hook input; leaving it unchanged", file=sys.stderr) + return 0 + payload = json.loads(raw) + except (ValueError, UnicodeError, RecursionError): + print("PIG token efficiency: invalid hook input; leaving it unchanged", file=sys.stderr) + return 0 + response = hook_response(payload, concise="--concise" in sys.argv, compact="--compact" in sys.argv) + if response is not None: + print(json.dumps(response, ensure_ascii=False)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/promptless_instruction_hub/token_efficiency_preview.py b/src/promptless_instruction_hub/token_efficiency_preview.py new file mode 100644 index 0000000..641edc7 --- /dev/null +++ b/src/promptless_instruction_hub/token_efficiency_preview.py @@ -0,0 +1,23 @@ +"""Non-mutating, exact diff preview for the same compactor used by the hooks.""" + +from __future__ import annotations + +import difflib +from pathlib import Path + +from promptless_instruction_hub.token_efficiency import compact_markdown + + +def preview_file(path: Path) -> str: + original = path.read_bytes().decode("utf-8") + compacted = compact_markdown(original) + before, after = len(original.encode("utf-8")), len(compacted.encode("utf-8")) + summary = f"{path}: {before} -> {after} UTF-8 bytes ({before - after} saved; not a token estimate)\n" + return summary + "".join( + difflib.unified_diff( + original.splitlines(keepends=True), + compacted.splitlines(keepends=True), + fromfile=str(path), + tofile=f"{path} (compacted)", + ) + ) diff --git a/tests/fixtures/token-efficiency/cases.json b/tests/fixtures/token-efficiency/cases.json new file mode 100644 index 0000000..2abb901 --- /dev/null +++ b/tests/fixtures/token-efficiency/cases.json @@ -0,0 +1,14 @@ +[ + {"id":"approval", "prompt":"In order to publish, require approval AND passing tests. Approval is absent; tests pass. Return only PUBLISH or WAIT.", "expected":"WAIT"}, + {"id":"negation", "prompt":"Never delete backups in order to free space. Space is low. Return only DELETE or KEEP.", "expected":"KEEP"}, + {"id":"exception", "prompt":"Skip all jobs with the exception of staging. Jobs are staging and production. Return only the job to run.", "expected":"staging"}, + {"id":"conditional", "prompt":"Retry in the event that status is 429. Status is 403. Return only RETRY or STOP.", "expected":"STOP"}, + {"id":"quantity", "prompt":"In order to retry, attempts must be below 3. Attempts equal 3. Return only RETRY or STOP.", "expected":"STOP"}, + {"id":"units", "prompt":"At this point in time, the limit is 12.5 GiB. Return only the exact limit including units.", "expected":"12.5 GiB"}, + {"id":"literal", "prompt":"Return exactly the quoted text: \"in order to\"", "expected":"in order to"}, + {"id":"code", "prompt":"Return exactly the inline code, without backticks: `if status != 200: stop()`", "expected":"if status != 200: stop()"}, + {"id":"prerequisite", "prompt":"In order to ship, both backup and rollback must be verified. Backup is verified; rollback is unknown. Return only SHIP or WAIT.", "expected":"WAIT"}, + {"id":"only", "prompt":"Only staging may restart in order to test. Production is failing. Return only RESTART or WAIT for production.", "expected":"WAIT"}, + {"id":"uncertainty", "prompt":"Deployment was queued; completion is unknown. In order to report success, completion must be verified. Return only SUCCESS or UNKNOWN.", "expected":"UNKNOWN"}, + {"id":"cause", "prompt":"The job stopped due to the fact that its token expired. Return only TOKEN or NETWORK for the stated cause.", "expected":"TOKEN"} +] diff --git a/tests/instruction_hub/test_token_efficiency.py b/tests/instruction_hub/test_token_efficiency.py new file mode 100644 index 0000000..4fdee0e --- /dev/null +++ b/tests/instruction_hub/test_token_efficiency.py @@ -0,0 +1,295 @@ +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest +import yaml +from pydantic import ValidationError + +from promptless_instruction_hub import token_efficiency +from promptless_instruction_hub.cli import main +from promptless_instruction_hub.compiler import build_hub, init_hub, verify_hub +from promptless_instruction_hub.models import TokenEfficiencyConfig +from promptless_instruction_hub.token_efficiency import compact_markdown, hook_response +from tests.config_helpers import enable_trace_ingestion + + +@pytest.mark.parametrize( + ("original", "expected"), + [ + ("In order to deploy, verify approval first.\n", "To deploy, verify approval first.\n"), + ("IN THE EVENT THAT tests fail, never deploy.", "IF tests fail, never deploy."), + ( + "Do not delete data in order to save 12.5 GiB unless approved.", + "Do not delete data to save 12.5 GiB unless approved.", + ), + ( + "Only retry in the event that status is 429; at most 3 times.", + "Only retry if status is 429; at most 3 times.", + ), + ("Skip all jobs with the exception of staging.", "Skip all jobs except staging."), + ("Keep plain prose concise.", "Keep plain prose concise."), + ("in order today", "in order today"), + ("İn order to ship", "İn order to ship"), + ("for the purpoſe of testing", "for the purpoſe of testing"), + ("éin order to ship", "éin order to ship"), + ], +) +def test_plain_prose_and_idempotence(original: str, expected: str) -> None: + assert compact_markdown(original) == expected + assert compact_markdown(expected) == expected + + +@pytest.mark.parametrize( + "text", + [ + "---\nname: x\ndescription: in order to route\n---\n", + "---\ndescription: in order to route\n", + "```python\nin order to\n```\n", + "~~~~text\nin order to\n~~~\nin order to\n~~~~\n", + "```text\nin order to\n", + " in order to\n\tin order to\n", + "> in order to\n", + "# In order to\n", + "Use `in order to` in order to match the exact string.\n", + 'Return "in order to" exactly.\n', + "Return 'in order to' exactly.\n", + "Use “in order to” literally.\n", + "[in order to](https://example.com/in-order-to)\n", + "Open /tmp/in order to/file.\n", + "in order to | due to the fact that\n", + "> quoted text\nin order to continue its lazy blockquote\n", + 'Return "multiline\n\nin order to\n\nvalue" exactly.\n', + "Return 'multiline\n\nin order to\n\nvalue' exactly.\n", + "Return “multiline\n\nin order to\n\nvalue” exactly.\n", + 'Return "unterminated\n\nin order to\n', + "in order to\n", + '{"value": "in order to"}\n', + ], +) +def test_literals_and_structure_are_opaque(text: str) -> None: + assert compact_markdown(text) == text + + +def test_fences_and_frontmatter_do_not_disable_following_prose() -> None: + original = "---\nname: x\n---\n\n```text\nin order to\n```\n\nIn order to test, run checks.\r\n" + assert compact_markdown(original) == original.replace("In order to test", "To test") + + +def test_hook_preserves_every_other_argument_and_permissions() -> None: + payload = { + "hook_event_name": "PreToolUse", + "tool_name": "Agent", + "tool_input": { + "prompt": "Never publish in order to bypass approval.", + "model": "sonnet", + "isolation": "worktree", + "run_in_background": True, + "unknown": {"future": [1, 2]}, + }, + } + response = hook_response(payload, concise=False, compact=True) + assert response == { + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "updatedInput": {**payload["tool_input"], "prompt": "Never publish to bypass approval."}, + } + } + assert "in order to" in json.dumps(payload) + + +@pytest.mark.parametrize( + "payload", + [ + None, + {"hook_event_name": []}, + {"hook_event_name": "PreToolUse", "tool_name": []}, + [], + {}, + {"hook_event_name": "UserPromptSubmit", "prompt": "in order to"}, + {"hook_event_name": "PreToolUse", "tool_name": "Bash", "tool_input": {"prompt": "in order to"}}, + {"hook_event_name": "PreToolUse", "tool_name": "Agent", "tool_input": {"prompt": 42}}, + {"hook_event_name": "PreToolUse", "tool_name": "Task", "tool_input": {"prompt": "already concise"}}, + ], +) +def test_unknown_and_unchanged_inputs_emit_nothing(payload: object) -> None: + assert hook_response(payload, concise=True, compact=True) is None + + +def test_switches_are_independent() -> None: + assert hook_response({"hook_event_name": "SessionStart"}, concise=False, compact=True) is None + assert hook_response({"hook_event_name": "SubagentStart"}, concise=True, compact=False) == { + "hookSpecificOutput": { + "hookEventName": "SubagentStart", + "additionalContext": token_efficiency.CONCISE_INSTRUCTION, + } + } + assert ( + hook_response( + {"hook_event_name": "PreToolUse", "tool_name": "Agent", "tool_input": {"prompt": "in order to"}}, + concise=True, + compact=False, + ) + is None + ) + + +def test_standalone_hook_invalid_input_and_emergency_disable() -> None: + runner = Path(token_efficiency.__file__) + for data in [b"not JSON", b"\xff", b"[" * 10000, b"a" * (token_efficiency.MAX_INPUT_BYTES + 1)]: + result = subprocess.run([sys.executable, str(runner), "--compact"], input=data, capture_output=True, check=True) + assert result.stdout == b"" + assert b"leaving it unchanged" in result.stderr + assert b"not JSON" not in result.stderr + result = subprocess.run( + [sys.executable, str(runner), "--concise"], + input=b'{"hook_event_name":"SessionStart"}', + capture_output=True, + check=True, + env={**os.environ, "PIG_TOKEN_EFFICIENT_MODE": "off"}, + ) + assert not result.stdout and not result.stderr + + +def _configure(root: Path, *, concise: bool, compact: bool) -> None: + path = root / "hub.yaml" + config = yaml.safe_load(path.read_text()) + config["token_efficiency"] = {"concise_output": concise, "compact_instructions": compact} + config["mcp"] = {"enabled": True} + path.write_text(yaml.safe_dump(config)) + + +def _skill(root: Path) -> str: + path = root / "assets/skills/sample" + path.mkdir(parents=True) + content = "---\nname: sample\ndescription: Use in order to test.\n---\n\nIn order to ship, never skip checks.\n" + (path / "SKILL.md").write_text(content) + (path / "reference.md").write_text("in order to preserve resource bytes\n") + (root / "plugins/pig.yaml").write_text("id: pig\nname: PIG\nincludes: [skill:sample]\n") + (root / "plugins/team.yaml").write_text("id: team\nname: Team\nincludes: [skill:sample]\n") + config_path = root / "hub.yaml" + config = yaml.safe_load(config_path.read_text()) + config["stable_plugins"].append("team") + config_path.write_text(yaml.safe_dump(config)) + return content + + +def test_opt_in_build_mcp_integrity_hook_merge_and_disable(tmp_path: Path) -> None: + init_hub(tmp_path, org="Promptless") + original = _skill(tmp_path) + build_hub(tmp_path) + assert (tmp_path / "dist/claude/pig/skills/sample/SKILL.md").read_text() == original + assert not (tmp_path / "dist/claude/pig/hooks/hooks.json").exists() + enable_trace_ingestion(tmp_path) + _configure(tmp_path, concise=True, compact=True) + before = (tmp_path / "assets/skills/sample/SKILL.md").read_bytes() + build_hub(tmp_path) + build_hub(tmp_path, check=True) + for target in ("claude", "codex", "gemini", "cursor"): + for plugin in ("pig", "team"): + root = tmp_path / "dist" / target / plugin + assert (root / "skills/sample/SKILL.md").read_text() == original.replace("In order to ship", "To ship") + assert (root / "skills/sample/reference.md").read_text() == "in order to preserve resource bytes\n" + assert (root / "hooks/pig-token-efficiency.py").exists() == (target == "claude" and plugin == "pig") + root = tmp_path / "dist/claude/pig" + hooks = json.loads((root / "hooks/hooks.json").read_text())["hooks"] + assert len(hooks["SessionStart"]) == 2 # managed trace hook is preserved + assert hooks["PreToolUse"][0]["matcher"] == "Agent|Task" + assert "UserPromptSubmit" not in hooks + result = subprocess.run( + hooks["PreToolUse"][0]["hooks"][0]["command"], + shell=True, + input=json.dumps( + {"hook_event_name": "PreToolUse", "tool_name": "Task", "tool_input": {"prompt": "in order to test"}} + ), + text=True, + capture_output=True, + check=True, + env={**os.environ, "CLAUDE_PLUGIN_ROOT": str(root)}, + ) + assert json.loads(result.stdout)["hookSpecificOutput"]["updatedInput"]["prompt"] == "to test" + manifest = json.loads((tmp_path / "dist/mcp/bundle.json").read_text()) + for asset in manifest["assets"]: + for file in asset["files"]: + data = (tmp_path / "dist/mcp" / file["path"]).read_bytes() + assert hashlib.sha256(data).hexdigest() == file["sha256"] + assert len(data) == file["size"] + mcp_content = (tmp_path / "dist/mcp/assets/skill/sample/SKILL.md").read_text() + assert mcp_content == original.replace("In order to ship", "To ship") + assert (tmp_path / "assets/skills/sample/SKILL.md").read_bytes() == before + _configure(tmp_path, concise=False, compact=False) + build_hub(tmp_path) + assert not (root / "hooks/pig-token-efficiency.py").exists() + assert (root / "skills/sample/SKILL.md").read_text() == original + assert "PreToolUse" not in json.loads((root / "hooks/hooks.json").read_text())["hooks"] + + +def test_verify_and_preview_never_modify_source(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: + init_hub(tmp_path, org="Promptless") + _skill(tmp_path) + _configure(tmp_path, concise=True, compact=True) + before = {p.relative_to(tmp_path): p.read_bytes() for p in tmp_path.rglob("*") if p.is_file()} + verify_hub(tmp_path) + assert main(["compact-preview", str(tmp_path / "assets/skills/sample/SKILL.md")]) == 0 + output = capsys.readouterr().out + assert "not a token estimate" in output + assert "-In order to ship" in output and "+To ship" in output + assert before == {p.relative_to(tmp_path): p.read_bytes() for p in tmp_path.rglob("*") if p.is_file()} + + +@pytest.mark.parametrize(("concise", "compact"), [(True, False), (False, True), (False, False)]) +def test_build_switches_are_independent(tmp_path: Path, concise: bool, compact: bool) -> None: + init_hub(tmp_path, org="Promptless") + original = _skill(tmp_path) + _configure(tmp_path, concise=concise, compact=compact) + build_hub(tmp_path) + root = tmp_path / "dist/claude/pig" + path = root / "hooks/hooks.json" + events = json.loads(path.read_text())["hooks"] if path.exists() else {} + assert ("SessionStart" in events) == concise + assert ("SubagentStart" in events) == concise + assert ("PreToolUse" in events) == compact + expected = original.replace("In order to ship", "To ship") if compact else original + assert (root / "skills/sample/SKILL.md").read_text() == expected + + +def test_agent_and_command_projections_preserve_sources_and_native_output(tmp_path: Path) -> None: + init_hub(tmp_path, org="Promptless") + _configure(tmp_path, concise=False, compact=True) + original = "---\ndescription: Use when shipping.\n---\n\nIn order to ship, require passing checks.\n" + agent = tmp_path / "assets/agents/reviewer.md" + agent.write_text(original) + agent.with_suffix(".asset.yaml").write_text( + "support:\n claude:\n mode: native\n codex:\n mode: agent-skill\n" + ) + command = tmp_path / "assets/commands/ship.md" + command.write_text(original) + (tmp_path / "plugins/pig.yaml").write_text("id: pig\nname: PIG\nincludes: [agent:reviewer, command:ship]\n") + build_hub(tmp_path) + for path in ( + "dist/codex/pig/skills/reviewer/SKILL.md", + "dist/claude/pig/skills/ship/SKILL.md", + "dist/mcp/projections/agent/reviewer/SKILL.md", + ): + assert (tmp_path / path).read_text().endswith("To ship, require passing checks.\n") + for path in ( + "dist/claude/pig/agents/reviewer.md", + "dist/mcp/assets/agent/reviewer/reviewer.md", + "dist/mcp/assets/command/ship/ship.md", + ): + assert (tmp_path / path).read_text() == original + manifest = json.loads((tmp_path / "dist/mcp/bundle.json").read_text()) + prompt = next(asset["prompt"] for asset in manifest["assets"] if asset["ref"] == "command:ship") + assert prompt["body"].endswith("To ship, require passing checks.\n") + assert agent.read_text() == command.read_text() == original + + +@pytest.mark.parametrize("value", [{"concise_output": "true"}, {"compact_instructions": 1}, {"unknown": True}]) +def test_config_rejects_ambiguous_switches(value: dict[str, object]) -> None: + with pytest.raises(ValidationError): + TokenEfficiencyConfig.model_validate(value)