ai: prepare for agentic workflows

.github/scripts/external-packages-license-check.go

@@ -59,7 +59,7 @@ func run() error {
 	}
 
 	// Check the internal dependencies of the external packages.
-	output, err := exec.Command("sh", "-c", "go list -deps -test ./public/*").Output()
+	output, err := exec.Command("sh", "-c", "go list -deps -test ./public/...").Output()
 	if err != nil {
 		return err
 	}

.kb/AGENTS.md

@@ -0,0 +1,45 @@
+# Preface
+
+This repository follows strict conventions for organizing agent-oriented knowledge documents, and this document is required reading for any agent that wants to read or write these.
+
+Read the top-level `.kb/agents.md` before continuing below.
+
+
+# Overview
+
+Every directory in this repository, including the root, may have its own `AGENTS.md` file and `.kb/` subdirectory. The AGENTS.md file provides a general view for the directory, while more specific information is found in `.kb/*.md` files with dashed lowercase names (e.g. `.kb/special-relativity.md`).
+
+The design of this structure has the following key goals:
+
+- **Mechanical** - Agents are the main actors reading and writing the knowledge base.
+- **Generic** - Benefits any agentic workflow, no matter the editor or platform.
+- **Distilled** - Avoids the use of verbose task or plan logs that pollute the context window.
+- **Hierarchical** - Avoids excessive information in a single place that also pollutes the context window.
+- **Human** - Information is readily available and reviewable in a friendly format.
+
+
+# Important
+
+- Read local `AGENTS.md` files upon navigating directories.
+- Keep the `.kb/*.md` files updated whenever there is something relevant to be documented or updated.
+- Follow the header conventions outlined below. Only the _Preface_ header is required, and the other headers should be omitted if empty or trivial.
+
+
+# Headers
+
+The following are the ONLY top-level headers allowed across the `.kb/*.md` files in this repository, to maintain semantic standardization across projects.
+Sub-headers are okay.
+
+- _Preface_ - A brief introduction outlining the scope and relevance of a specific `*.md` file. This section MUST be at the top of every `.kb/*.md` file so agents can easily grep for it, and the last line of this section MUST be "Read the top-level `.kb/agents.md` file before continuing below." so rules are followed.
+- _Overview_ - High-level summary of the directory, subsystem or knowledge base layout at large. Do NOT use this to list files or directories.
+- _Important_ - Essential directives for the agent outlining critical constraints, behaviors, or rules.
+- _Headers_ - Global registry of header definitions, uniquely hosted at the root `.kb/agents.md` (this file). Do NOT use this header in any other document.
+- _Architecture_ - Structural design details or boundary explanations for a given component. Only use this for software architecture concepts, NOT for directory outlining. Also avoid using this as a code reference (keep that in the code itself).
+- _Directory_ - Only in `AGENTS.md` files, it briefly outlines the contents and structure of the directory the `AGENTS.md` file is in, and potentially nested small directories that do not justify their own `AGENTS.md` file. For `.kb/*.md` files, use _Documents_ instead.
+- _Documents_ - Only in `AGENTS.md` files, it outlines the content of `.kb/*.md` files and also immediately nested `AGENTS.md` child files, to aid agent navigation. It MUST be the last section in the file. It's okay to also mention such filenames inline when they are relevant elsewhere in the text.
+
+For the _Directory_ and _Documents_ listings, format items as a dashed list starting with the file or directory name surrounded by backticks, a dash, and then a brief description:
+```
+- `filename` - Terse summary.
+```
+

AGENTS.md

@@ -0,0 +1,33 @@
+# Preface
+
+This repository hosts Chisel, a software tool for carving and cutting Debian packages. This `AGENTS.md` is the root index into the dispersed knowledge base; read it to orient yourself, then follow the links below for details.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+
+# Directory
+
+- `cmd/` - Binary entry points for the `chisel` application.
+- `internal/` - Core internal packages: slice orchestration, setup, extraction, archive, cache, and supporting utilities.
+- `public/` - Public API packages for external consumers.
+- `snap/` - Snap packaging configuration for Chisel.
+- `tests/` - End-to-end integration test suite.
+- `docs/` - Documentation assets.
+- `go.mod` - Go module definition.
+- `go.sum` - Go dependency checksums.
+- `spread.yaml` - Spread test configuration.
+- `workshop.yaml` - Workshop configuration.
+- `.golangci.yaml` - golangci-lint configuration.
+- `README.md` - Project readme.
+- `SECURITY.md` - Security policy.
+- `LICENSE` - Project license.
+
+
+# Documents
+
+- `.kb/agents.md` - General rules for the knowledge base reading and writing.
+- `cmd/AGENTS.md` - CLI entry points.
+- `internal/AGENTS.md` - Core internal packages.
+- `public/AGENTS.md` - Public API packages.
+- `snap/AGENTS.md` - Snap packaging.
+- `tests/AGENTS.md` - Integration testing.

cmd/AGENTS.md

@@ -0,0 +1,25 @@
+# Preface
+
+This document serves as the local knowledge base index for the `cmd/` directory. It outlines the scope and purpose of the entry points for the `chisel` binary.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `cmd/` directory contains the main binary entry points for the `chisel` application. Instead of housing core business logic, this directory is responsible for initializing the application, wiring up the CLI subsystem, and handling version information. Core logic is deferred to the packages in `internal/`.
+
+# Directory
+
+- `chisel/` - Contains the primary executable entry point for the `chisel` application.
+- `chisel/main.go` - Application bootstrap and CLI initialization.
+- `chisel/cmd_cut.go` - Implements the `cut` command, the primary operation that extracts slices into a target root filesystem.
+- `chisel/cmd_find.go` - Implements the `find` command for searching available slices by name or package.
+- `chisel/cmd_info.go` - Implements the `info` command for displaying detailed metadata about specific slices.
+- `chisel/cmd_debug.go` - Implements the `debug` subcommand group for internal diagnostics.
+- `chisel/cmd_debug_check_release_archives.go` - Implements `debug check-release-archives`, which validates the release with packages content from the archive.
+- `chisel/cmd_version.go` - Implements the `version` command.
+- `chisel/cmd_help.go` - Implements the `help` command.
+- `chisel/helpers.go` - Shared CLI utilities used across commands.
+- `chisel/log.go` - Logging setup for the CLI layer.
+- `mkversion.sh` - Shell script used for generating version information during builds.
+- `version.go` - Application version data structures and variables.

internal/AGENTS.md

@@ -0,0 +1,64 @@
+# Preface
+
+This is the index file for the `internal/` directory's knowledge base. It provides context about the core internal packages of Chisel, encompassing slice orchestration, package setup, extraction, archive fetching, caching, filesystem operations, manifest generation, and supporting utilities.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `internal/` directory houses the core business logic, components, and utilities of Chisel. It coordinates slice selection, dependency resolution, package fetching, file extraction, filesystem mutations, and manifest output. These packages are not part of Chisel's public API and must not be imported by external consumers.
+
+# Directory
+
+- `slicer/` - Main orchestrator for a Chisel run. Receives a slice selection, drives all other internal packages (setup, archive, cache, deb, fsutil, scripts, manifestutil) to completion, and writes the final filesystem and manifest.
+- `setup/` - Parses chisel-releases YAML definitions into the `Release` model and resolves slice dependencies, path conflicts, and package contention.
+- `deb/` - Debian package utilities: data tarball access, version comparison, and architecture handling.
+- `tarball/` - Extracts selected files from a package data tarball into a target directory.
+- `archive/` - Manages remote Ubuntu package archive sources over HTTP/HTTPS.
+- `cache/` - Content-addressable on-disk store keyed by SHA256 digest, with time-based eviction.
+- `fsutil/` - Core filesystem operations for writing files, directories, and symlinks into the target root filesystem.
+- `manifestutil/` - Generates the Chisel manifest.
+- `scripts/` - Executes Starlark mutation scripts defined in slice definitions.
+- `control/` - Parser for Debian control files (the metadata sections embedded in `.deb` archives).
+- `strdist/` - String distance and glob matching utilities. Implements a configurable edit-distance algorithm (`Distance`) with pluggable cost functions, and a `GlobPath` function that uses that algorithm to match file paths against patterns supporting `?`, `*`, and `**` wildcards.
+- `pgputil/` - Decodes and validates PGP signatures on package archive metadata. Wraps `golang.org/x/crypto/openpgp`.
+- `testutil/` - Shared test helpers used across unit tests: mock archive builders, composable content checkers, file presence and permission validators, tree dumpers, and permutation utilities.
+- `apacheutil/` - Shared slice-naming utilities (`SliceKey`, name-format regexps, `ParseSliceKey`). The "apache" prefix signals that this package carries an Apache-2.0 license, which is required because it is a transitive dependency of the `public/` packages.
+- `apachetestutil/` - Test helpers for reading manifest contents (`DumpManifestContents`), carrying the same Apache-2.0 license requirement as `apacheutil/` because it is depended on by tests in the `public/` packages.
+
+# Architecture
+
+Chisel's internal packages form a directed dependency chain driven by `slicer/`:
+
+```mermaid
+flowchart LR
+    slicer["slicer<br/>Orchestrator"]
+
+    subgraph logic["Core Logic"]
+        setup["setup<br/>Release parsing, dep resolution,<br/>conflict detection"]
+        archive["archive<br/>Ubuntu archive HTTP client"]
+        manifestutil["manifestutil<br/>Manifest writer"]
+        scripts["scripts<br/>Starlark mutations"]
+    end
+
+    subgraph base["Extraction & Storage"]
+        deb["deb<br/>.deb file extractor"]
+        tarball["tarball<br/>Data tarball extractor"]
+        fsutil["fsutil<br/>Filesystem writer"]
+        cache["cache<br/>Content-addressable store"]
+    end
+
+    subgraph util["Utilities"]
+        control["control<br/>Debian control parser"]
+        pgputil["pgputil<br/>PGP verification"]
+        strdist["strdist<br/>Glob & distance matching"]
+    end
+
+    slicer --> setup & archive & tarball & fsutil & scripts & manifestutil
+    setup --> archive & deb & cache & strdist
+    manifestutil --> archive & setup
+    archive --> cache & control & pgputil & deb
+    tarball --> deb & fsutil & strdist
+    deb --> fsutil & strdist
+    scripts --> fsutil
+```

public/AGENTS.md

@@ -0,0 +1,14 @@
+# Preface
+
+This document describes the scope of the `public/` directory, which contains Chisel's public API packages intended for consumption by external tools.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `public/` directory houses the two packages that form Chisel's stable public contract. These packages define the on-disk format and data schema for the Chisel manifest, enabling third-party tools such as SBOM generators and vulnerability scanners to consume Chisel output without depending on internal packages.
+
+# Directory
+
+- `jsonwall/` - Defines and implements the "jsonwall" database format: a ZSTD-compressed text file with one JSON object per line, with fields sorted for efficient search and iteration. This is the storage format used for the Chisel manifest.
+- `manifest/` - Defines the manifest entry schema (schema version 1.0), including the `Package`, `Slice`, and `File` record types with their fields (`Kind`, `Name`, `Version`, `Digest`, `Arch`, etc.). Integrates with `jsonwall` for serialization and deserialization.

snap/AGENTS.md

@@ -0,0 +1,9 @@
+# Preface
+
+This document describes the scope of the `snap/` directory within the Chisel repository, providing context for automated agents navigating and modifying the snapcraft configuration.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Directory
+
+- `snapcraft.yaml` - The primary manifest file defining the snap package. It configures classic confinement, utilizes the Go plugin, and outlines the build steps which depend on `cmd/mkversion.sh` for version injection.

tests/AGENTS.md

@@ -0,0 +1,30 @@
+# Preface
+
+This file serves as the local knowledge base index for the `tests/` directory. It outlines the scope and procedures for end-to-end integration testing within the Chisel repository.
+
+Read the top-level `.kb/agents.md` file before continuing below.
+
+# Overview
+
+The `tests/` directory contains the integration test suite for the project. These tests exercise the actual `chisel` binary against real or mock Ubuntu package archives, verifying end-to-end behaviors such as slice extraction, manifest generation, and error handling. The suite is built on the [Spread](https://github.com/canonical/spread) framework for multi-system test execution.
+
+# Important
+
+- **Execution**: Integration tests are run with `spread`, not with `go test`. Each test scenario is a subdirectory containing a `task.yaml` with shell-based assertions.
+- **No build tags**: Unlike Go-based integration test suites, these tests require no `//go:build` directives. They are entirely shell-driven.
+- **Pre-built binary**: Spread compiles and provisions the `chisel` binary as part of the test environment setup defined in `spread.yaml` at the repository root.
+
+# Directory
+
+- `basic/` - Core slice extraction scenario verifying that files are correctly written to the target root filesystem and that mutation scripts are applied.
+- `find/` - Tests for the `chisel find` command, covering search by slice name and package.
+- `info/` - Tests for the `chisel info` command, verifying detailed slice metadata output.
+- `debug-check-release-archives/` - Tests for the `chisel debug check-release-archives` command, validating archive configuration correctness.
+- `pro-archives/` - Tests covering Ubuntu Pro subscription archive support (fips, fips-updates, esm-apps, esm-infra).
+- `use-a-custom-chisel-release/` - Tests the ability to override the default chisel-releases with a custom release tree.
+- `unmaintained/` - Edge-case tests for packages whose support window has ended.
+- `unstable/` - Edge-case tests for packages from unstable or unsupported releases.
+
+# Architecture
+
+Each test scenario is a directory containing a `task.yaml` that defines the test steps as shell commands with assertions (typically `grep`-based). Spread handles multi-system provisioning, binary deployment, and test execution across different Ubuntu releases as configured in the root `spread.yaml`.