feat(bundle): images a bundle carries as docker save archives, loaded rather than built or pulled

AGENTS.md

@@ -117,7 +117,7 @@ and deserialized through a registry.
 | `openenv/` | `build_app(bundle)`: the [OpenEnv](https://github.com/openenv/OpenEnv) server for a bundle's tasks, each session playing one episode through the Episode API, with the envs' MCP tools, an added `end_turn` tool and the task's verifier score as the reward. It prepares bundles as `agent-env run` does (`prepare_bundle(played=True)`, without the agent the client plays) and keeps namespace routing on for the app's life. `openenv` comes with the `openenv` extra, a floor with no cap; this package and `training/trl.py` are the only ones that import it. Each task's description in the Task API names its `toolset` (the played agent's envs, role, and whether it gets `bash`). |
 | `training/` | One module per outside trainer, each behind its own extra, and `_tools.py`, which turns an env's tools into typed, documented Python methods from their JSON Schema, as transformers' `get_json_schema` reads them. `trl.py` (`trl` extra): `AgentEnv(url)` over a server `openenv/` makes, with `dataset` (a row per task) and `factory` (TRL's `environment_factory`: a generated class per toolset whose methods are the tools, a background `reset` and an owned `get_reward`); `AgentEnv.local(bundle)` serves the bundle in process. |
 | `runner/` | The `[runner]` seam: `Runner.submit()` returns a `RunHandle` (`run_id`, `instance_id`) before the run starts; `LocalRunner` is built in. `runner/store.py` is the run index every runner writes and the explorer reads. |
-| `bundle/` | A bundle is a folder of entities (`envs/`, `agents/`, `artifacts/`, `skills/`, `tasks/<name>.json`, `evals/<name>.toml`) that runs without a registry: `agent-env run` plans it, writes it into the `@local` namespace (reusing each version whose inputs haven't changed) and runs its evals or tasks; `agent-env bundle init` writes a stored task or eval as one, with every env, agent and artifact it uses in its own folder (each image as its build context, pruned to what its Dockerfile reads and pinned to the Python and npm versions of the stored image) so it runs with no store; `agent-env bundle register` writes a folder into the configured stores under registry ids (`<prefix><name>`), each task and eval naming the versions it reads and each image it builds pushed, through a ledger in the target store so registering again writes only what changed. |
+| `bundle/` | A bundle is a folder of entities (`envs/`, `agents/`, `artifacts/`, `skills/`, `tasks/<name>.json`, `evals/<name>.toml`) that runs without a registry: `agent-env run` plans it, writes it into the `@local` namespace (reusing each version whose inputs haven't changed) and runs its evals or tasks; an image is built from a Dockerfile, pinned from a registry (`{ ref = "<image>" }`), or loaded from a `docker save` archive in its folder (`{ archive = "<path>" }`, or `archive =` in an image folder's `artifact.toml`), stored as its tar.gz naming no image (so a load never replaces an image the machine holds under a name the archive gives) under a tag its digest gives, with the Dockerfile it names kept beside it as its build context, which local Docker and VM sandboxes load and a sandbox that only pulls refuses; `agent-env bundle init` writes a stored task or eval as one, with every env, agent and artifact it uses in its own folder (each image as its build context, pruned to what its Dockerfile reads and pinned to the Python and npm versions of the stored image) so it runs with no store; `agent-env bundle register` writes a folder into the configured stores under registry ids (`<prefix><name>`), each task and eval naming the versions it reads and each image it builds pushed, through a ledger in the target store so registering again writes only what changed. |
 | `eval/` | `Eval` (`eval`), a versioned list of task references, and its store. |
 | `plugins/` | Entry-point plugins (see Extension points): the group names, `inventory()`, which `agent-env plugin list/show/check` report, `load_failures` and `settings`. The top level is public; the underscored modules are internal. |
 | `explorer/` | Optional local web UI: `agent-env up`, binds loopback `:8234`. |

src/agent_env/artifact/artifacts/docker_image.py

@@ -4,6 +4,7 @@
 
 import asyncio
 import gzip
+import hashlib
 import logging
 import posixpath
 import re
@@ -25,6 +26,7 @@
 from agent_env.store.image_store.registry_api import NoSuchImageError, pin_digest, pinned_from
 from agent_env.utils.docker.context import BuildContext
 from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, build_image, normalize_platform
+from agent_env.utils.docker.untag import write_untagged
 
 logger = logging.getLogger(__name__)
 
@@ -282,6 +284,39 @@ def put_context(cls, id: str, *, description: str, context_path: str, dockerfile
             source_digest=source_digest,
         ))
 
+    @classmethod
+    def put_archive(cls, id: str, *, description: str, archive_path: str, context_path: str | None = None,
+                    dockerfile_path: str | None = None, platform: str | None = None) -> "DockerImageArtifact":
+        """Register the image ``docker save`` wrote to ``archive_path``, stored as its tar.gz, which a deploy loads, so
+        no Docker is needed here; with the build context at ``context_path`` kept beside it when given. It's stored
+        naming no image, so a load tags nothing the archive names; its ``image_name``,
+        ``local/<slug>-<hash>:<archive>``, the same for the same archive, names no registry, and each load tags the
+        image with it. Raises ValueError when the archive isn't one docker save writes."""
+        from agent_env.artifact.store import get_artifact_store
+        from agent_env.config import get_config
+
+        context = _context_of(Path(context_path), dockerfile_path) if context_path else None
+        platform = normalize_platform(platform) if platform else None
+        store = get_artifact_store()
+        prefix = store.attempt_prefix("docker_image", id)
+        objects = get_config().get_object_store_to_write(prefix, id)
+        digest = _sha256(Path(archive_path))
+        with tempfile.TemporaryDirectory(prefix="agent-env-archive-") as tmp:
+            untagged = Path(tmp) / "image.tar.gz"
+            with open(archive_path, "rb") as saved, untagged.open("wb") as out:
+                write_untagged(saved, out, gzip_level=1)
+            tar_gz_object_url = objects.put_file_at(f"{prefix}{fs_safe(id)}.tar.gz", str(untagged), "application/gzip")
+        return cls.put_tar(
+            id,
+            description=description,
+            image_name=f"{local_image_repository(id)}:{digest[:_SOURCES_TAG_CHARS]}",
+            tar_gz_object_url=tar_gz_object_url,
+            build_context_object_url=_upload_context(objects, prefix, context) if context else None,
+            dockerfile_path=context.dockerfile if context else None,
+            platform=platform,
+            source_digest=context.source_digest(platform) if context else None,
+        )
+
     @staticmethod
     def context_identity(id: str, context_path: str, dockerfile_path: str,
                          platform: str | None = DEFAULT_BUILD_PLATFORM) -> tuple[str, str]:
@@ -320,9 +355,11 @@ def stored_under_build_tag(self) -> bool:
         return bool(self.tar_gz_object_url) and not names_registry(self.image_name)
 
     def build_tag_reason(self) -> str:
-        """Why a sandbox elsewhere can't pull an image ``stored_under_build_tag``."""
-        return (f"{self.id!r} v{self.version} is stored as a tar.gz under {self.image_name!r}, which names no registry: "
-                "it's the tag the machine that built it gave it, which a sandbox elsewhere can't pull")
+        """Why a sandbox elsewhere can't pull an image ``stored_under_build_tag``; one not yet written has no
+        version."""
+        named = f"{self.id!r} v{self.version}" if self.version else repr(self.id)
+        return (f"{named} is stored as a tar.gz under {self.image_name!r}, which names no registry: it's a tag only a "
+                "machine that loads the tar.gz has, which a sandbox elsewhere can't pull")
 
     def rebuilt_for(self, platform: str | None) -> bool:
         """Whether a VM sandbox of ``platform`` builds this image from its build context rather than load its tar.gz,
@@ -674,6 +711,14 @@ def _context_image_name(id: str, source_digest: str) -> str:
     return f"{local_image_repository(id)}:{source_digest.removeprefix('sha256:')[:_SOURCES_TAG_CHARS]}"
 
 
+def _sha256(path: Path) -> str:
+    hashed = hashlib.sha256()
+    with path.open("rb") as file:
+        while chunk := file.read(1 << 20):
+            hashed.update(chunk)
+    return hashed.hexdigest()
+
+
 def _upload_context(objects, prefix: str, context: BuildContext) -> str:
     """Write ``context`` under ``prefix`` as a tar.gz and return its object URL."""
     with tempfile.TemporaryDirectory(prefix="agent-env-context-") as tmp:

src/agent_env/bundle/authoring.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import os
+import posixpath
 import shutil
 import stat
 import tempfile
@@ -13,19 +14,23 @@
 from pathlib import Path
 from typing import Any, NoReturn, TypeVar, overload
 
+from agent_env.a2a_agent import A2AAgent
 from agent_env.artifact.artifact import Artifact
 from agent_env.artifact.artifacts.docker_image import DockerImageArtifact
+from agent_env.artifact.registry import canonical_type, get_artifact_registry
 from agent_env.entity.refs import EntityKind, parse_toml_ref
 from agent_env.env.env import Env
 from agent_env.env.envs._deployment import provider_refusal
+from agent_env.env.registry import get_env_registry
+from agent_env.eval.eval import Eval
 from agent_env.providers.env_providers.constants import GATEWAY_SERVICE_NAMES
 from agent_env.providers.env_providers.env_gateway_provider import EnvironmentGatewayProvider, environment_name_problem
 from agent_env.providers.env_providers.env_provider import _env_provider_class
 from agent_env.utils.card_naming import card_names_in_files
 from agent_env.utils.paths import os_reason
 
 from ._fs import relative, show, with_article
-from .parse import CONFIG_FILES, LEAVINGS, NAMED_BY, Bundle, BundleEntry, BundleError
+from .parse import CONFIG_FILES, LEAVINGS, NAMED_BY, Bundle, BundleEntry, BundleError, BundleKind
 
 A = TypeVar("A", bound=Artifact)
 E = TypeVar("E", bound=Env)
@@ -248,12 +253,40 @@ def _fail(self, message: str) -> NoReturn:
 
 def entry_files(bundle: Bundle, entry: BundleEntry, *, empty_ok: bool = False) -> dict[str, Path]:
     """The files in an entry's folder, keyed by POSIX path in NFC and sorted: regular files, dot files
-    included, through links that stay inside the bundle, less what the OS or Python leaves behind and
-    the entry's own toml. Raises BundleError listing every problem found, a folder with no files among
-    them unless ``empty_ok``."""
+    included, through links that stay inside the bundle, less what the OS or Python leaves behind, the
+    entry's own toml and the image archives it names. Raises BundleError listing every problem found, a
+    folder with no files among them unless ``empty_ok``."""
     return _Walk(bundle, entry, empty_ok).files()
 
 
+def entity_class(entry: BundleEntry) -> type | None:
+    """The class that writes ``entry``: its env type's, its artifact type's, the agent's or the eval's; None for a type
+    nothing registers."""
+    if entry.kind is BundleKind.ENV:
+        return get_env_registry().get(entry.type)
+    if entry.kind is BundleKind.AGENT:
+        return A2AAgent if entry.type == A2AAgent.type else None
+    if entry.kind is BundleKind.EVAL:
+        return Eval if entry.type == Eval.type else None
+    return get_artifact_registry().get(canonical_type(entry.type))
+
+
+def entry_archives(entry: BundleEntry) -> frozenset[str]:
+    """The image archives ``entry``'s toml names, as POSIX paths in its folder in NFC: an image folder's ``archive``,
+    and the ``archive`` of each image key its type declares, as the resolver reads them. What an image loads, which no
+    build context or file the entry writes holds."""
+    config = entry.config if isinstance(entry.config, dict) else {}
+    tables = [config] if entry.kind is BundleKind.ARTIFACT and canonical_type(entry.type) == "docker_image" else []
+    for ref in getattr(entity_class(entry), "toml_refs", ()):
+        if ref.artifact_type == "docker_image" and "[]" not in ref.path:
+            value: Any = config
+            for key in ref.path.split("."):
+                value = value.get(key) if isinstance(value, dict) else None
+            tables.append(value)
+    return frozenset(unicodedata.normalize("NFC", posixpath.normpath(table["archive"])) for table in tables
+                     if isinstance(table, dict) and isinstance(table.get("archive"), str))
+
+
 def build_context_files(bundle: Bundle, entry: BundleEntry) -> dict[str, Path]:
     """What an image built from an entry's folder is built from: its ``entry_files`` and its own toml, which
     the Dockerfile can copy too."""
@@ -337,6 +370,8 @@ def files(self) -> dict[str, Path]:
             self._problem(top, f"{how} {show(os.path.realpath(top))}, outside the bundle; {_BRING_IT_IN}")
         else:
             self._walk(top)
+        for archive in entry_archives(self.entry):
+            self.found.pop(archive, None)
         if not self.found and not self.problems and not self.empty_ok:
             self._problem(top, f"has no files to write ({', '.join(sorted(LEAVINGS))} don't count)")
         if self.problems:

src/agent_env/bundle/ledger.py

@@ -32,9 +32,9 @@
 from agent_env.task.store import TASKS_COLLECTION
 
 from .authoring import build_context_files
-from .parse import BundleKind
+from .parse import Bundle, BundleKind
 from .plan import Plan, Write, artifact_writer, env_folder_writer, env_writer, folder_files, unpinned_store_refs
-from .resolve import BuiltImage, BundleImage, ReferencedImage
+from .resolve import ArchivedImage, BuiltImage, BundleImage, ReferencedImage
 
 LEDGER_COLLECTION = "bundle_ledger"
 # Bumped by hand when what a digest covers changes, or a writer's output changes enough that every
@@ -133,7 +133,9 @@ def digest(self, write: Write, needs: Mapping[tuple[str, str], int]) -> Digest |
                   "files": {}, "needs": {}, "store_refs": {}}
         from_folder = write.kind in (BundleKind.ARTIFACT, BundleKind.SKILL) or (
             write.kind is BundleKind.ENV and env_folder_writer(get_env_registry().get(_type(write))))
-        if from_folder and not isinstance(write.source, ReferencedImage):
+        if isinstance(write.source, ArchivedImage):
+            inputs["files"] = _archived_files(self._plan.bundle_of(write.source.entry), write.source)
+        elif from_folder and not isinstance(write.source, ReferencedImage):
             built = isinstance(write.source, BuiltImage)
             listing = build_context_files if built else folder_files
             for key, path in listing(self._plan.bundle_of(write.source.entry), write.source.entry).items():
@@ -255,7 +257,11 @@ def _config(write: Write) -> Any:
     source = write.source
     if not isinstance(source, BundleImage):
         return source.config
-    config = {"dockerfile": source.dockerfile} if isinstance(source, BuiltImage) else {"ref": source.ref}
+    if isinstance(source, ArchivedImage):
+        config = {key: value for key, value in (("archive", source.archive), ("dockerfile", source.dockerfile),
+                                                ("platform", source.platform)) if value is not None}
+    else:
+        config = {"dockerfile": source.dockerfile} if isinstance(source, BuiltImage) else {"ref": source.ref}
     if source.description is not None:  # only an image folder's, so an image key's digest is as before
         config["description"] = source.description
     return config
@@ -327,6 +333,16 @@ def _sha256(data: bytes) -> str:
 _USUAL_MODE = 0o644
 
 
+def _archived_files(bundle: Bundle, image: ArchivedImage) -> dict[str, str]:
+    """What an archived image is written from: its archive, and its build context's files, with their modes, when it
+    keeps one."""
+    files = {image.archive: _file_sha256(image.entry.path / image.archive)}
+    if image.dockerfile is not None:
+        for key, path in build_context_files(bundle, image.entry).items():
+            files[key] = _file_sha256(path) + _mode(path)
+    return files
+
+
 def _mode(path: Path) -> str:
     mode = stat.S_IMODE(path.stat().st_mode)
     return "" if mode == _USUAL_MODE else f"+{mode:o}"

src/agent_env/bundle/materialize.py

@@ -17,6 +17,7 @@
 from __future__ import annotations
 
 import copy
+import functools
 from collections.abc import Callable, Iterator, Mapping
 from contextlib import contextmanager, nullcontext
 from dataclasses import dataclass
@@ -42,7 +43,7 @@
 from .ledger import Ledger, locked_ids, materializing
 from .parse import BundleError, BundleKind
 from .plan import Plan, Write, artifact_writer, env_writer, unpinned_store_refs
-from .resolve import BuiltImage, BundleImage, ReferencedImage, build_step
+from .resolve import ArchivedImage, BuiltImage, BundleImage, ReferencedImage, build_step
 
 
 @dataclass(frozen=True)
@@ -153,6 +154,8 @@ def through_ledger(write: Write, write_fn: Callable[[], int]) -> None:
 def _write_entity(plan: Plan, write: Write, here: bool, on_build: Callable[[Write], None] | None) -> int:
     if isinstance(write.source, ReferencedImage):
         return _write_referenced_image(plan, write)
+    if isinstance(write.source, ArchivedImage):
+        return _write_archived_image(plan, write)
     if isinstance(write.source, BuiltImage):
         if write.id in plan.published:
             return _write_published_image(plan, write)
@@ -253,6 +256,21 @@ def _write_referenced_image(plan: Plan, write: Write) -> int:
         raise BundleError([f"{_path(plan, write)}: {e}"]) from None
 
 
+def _write_archived_image(plan: Plan, write: Write) -> int:
+    """Register the image an entry's archive holds, stored as its tar.gz, which each deploy loads, with the build
+    context its Dockerfile names kept beside it when it names one."""
+    image, entry = write.source, write.source.entry
+    put = functools.partial(DockerImageArtifact.put_archive, write.id, description=_image_description(plan, write),
+                            archive_path=str(entry.path / image.archive), platform=image.platform)
+    try:
+        if image.dockerfile is None:
+            return put().version
+        with staged_build_context(plan.bundle_of(entry), entry, image.dockerfile) as staged:
+            return put(context_path=str(staged.path), dockerfile_path=staged.dockerfile).version
+    except (ValueError, RuntimeError, OSError) as e:  # a context it can't write, or the object store
+        raise BundleError([f"{_path(plan, write)}: {e}"]) from None
+
+
 def _latest_image_name(image_id: str) -> str | None:
     """The image name ``image_id``'s latest version records, when that version is an image: a folder can change type."""
     try:
@@ -269,6 +287,8 @@ def _image_description(plan: Plan, write: Write) -> str:
         return image.description
     if isinstance(image, BuiltImage):
         return f"built from {_path(plan, write)}/{image.dockerfile}"
+    if isinstance(image, ArchivedImage):
+        return f"loaded from {_path(plan, write)}/{image.archive}"
     return f"referenced by {_path(plan, write)}"