Retain Jira originals and enable native project task tracking (JVNAUTOSCI-2737)

AGENTS.md

@@ -407,6 +407,13 @@ mechanism, or grant of merge authority.
 
 ## 8. Task and repository discipline
 
+- For a project represented in Von, read its task-project `writer` before
+  choosing the tracking surface. When it is `von`, use native task tools and
+  the associated collections for creation, updates and completion; legacy Jira
+  keys remain lookup aliases. When it is `jira`, keep Jira authoritative and
+  treat imported tasks as migration copies. Keep Jira access available for
+  provenance and migration recovery. See the
+  [Jira migration guide](docs/engineering/workflow_first_jira_to_von_migration_jvnautosci_1223.md).
 - Use New Zealand English spelling by default.
 - Use the native shell for the host: PowerShell on Windows; `zsh`/`sh` on
   macOS and Linux, except when deliberately invoking or testing another shell.

docs/engineering/workflow_first_jira_to_von_migration_jvnautosci_1223.md

@@ -1,6 +1,137 @@
-# JVNAUTOSCI-1223 Workflow-First Jira -> Von Migration
-
-Date: 2026-02-21
+# Jira to Von task migration and project cutover
+
+Last updated: 10 September 2026. Current delivery decision:
+[JVNAUTOSCI-2737](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2737).
+
+## Full source retention and project cutover
+
+The ordinary task projection and the retained Jira original serve different
+purposes. A task count or a successful import does not establish full content
+retention. `--preserve-source` keeps complete source documents, rendered fields,
+field schemas/names, paginated comments, worklogs, history, properties, remote
+links, votes/watchers, and original attachment bytes in immutable, hashed Von
+file copies. Native task fields support day-to-day work; unsupported fields
+remain retrievable in the original. Capture errors remain explicit and block a
+full retention claim.
+
+Each source project has a stable Von project record and an associated primary
+task collection. Board and saved-filter snapshots can be associated with every
+project they cover. Collections preserve their source definitions and selected
+issue keys; native additions use explicit membership. They do not execute JQL
+or reproduce Jira board automation. Project descriptions, properties, roles,
+components, versions, workflow/status and security configuration are retained
+as source information. Source sharing settings do not grant access in Von.
+
+### Run and recover an import
+
+Use the canonical runner with an explicitly configured local operator:
+
+```sh
+pdm run python scripts/run_jira_task_migration.py \
+  --project-key PROJECT --actor-concept-id '#V#operator' \
+  --preserve-source --include-done --passes 1 --batch-size 25 \
+  --resume --report-path /private/audit/PROJECT.json
+```
+
+Supply `--organisation-concept-id` only for the intended organisational project.
+Other projects can remain private to the operator. Source project IDs and site
+identity distinguish projects; stable issue IDs and previous keys preserve
+lookup after source moves. All issue types and terminal statuses are included
+by this command. Do not use `--only-missing` when existing projections still
+need original-content retention.
+
+For attachments above the Jira proxy's default 10 MiB retrieval limit, set
+`VON_INTERNAL_MCP_JIRA_ATTACHMENT_MAX_SIZE_BYTES=104857600` in the migration
+process environment. This explicitly permits retrieval up to 100 MiB; larger
+files remain reported gaps requiring a separate bounded capture.
+
+Completed batches are checkpointed atomically. Repeating the command resumes a
+matching source snapshot. If Jira has changed, use a fresh report path for the
+delta and reconcile it against fresh full discovery. Reruns retain native edits
+using the previous import projection, report conflicts, preserve prior source
+archives, and deduplicate source activity. Unresolved hierarchy/link targets
+can be repaired from retained import rows after all projects have imported.
+Never infer deletion authority from a missing source issue.
+
+`scripts/capture_jira_project_sources.py` captures project and site source
+configuration through named Jira MCP resources and associates board/filter
+collections. Inspect its report for unsupported or inaccessible resources.
+Automation exports, cloud exports, app-owned data, forms and linked external
+documents require explicit capture or an evidence-backed scope disposition;
+REST access failures are not proof that a resource is empty. Keep these private
+source artefacts outside the repository.
+
+### Native use and actor binding
+
+The Tasks panel supports project/collection selection, ordinary task edits,
+hierarchy, comments, attachments, worklogs and transitions. Imported tasks link
+to their retained original without needing Jira. Native MCP tools expose the
+same canonical services, including `task_list_projects`, `task_get_project`,
+`task_get_collection`, `task_get`, `task_search` and `task_get_source_archive`.
+`task_get` accepts legacy Jira keys as well as Von task concept IDs.
+
+A local stdio connection can bind the trusted operator once in its server
+environment:
+
+```toml
+[mcp_servers.vontology.env]
+VON_MCP_TASK_ACTOR_CONCEPT_ID = "#V#operator"
+VON_MCP_TASK_ORGANISATION_CONCEPT_ID = "#V#organisation"
+```
+
+The organisation is optional. This is local operator configuration, not a
+client-supplied identity claim. It applies only to task tools, retains canonical
+access checks and the configured write profile, rejects a conflicting actor
+payload, and never replaces an inherited actor context. Browser and internal
+workflow calls continue to derive identity from their trusted session context.
+
+### Reconciliation, backup and writer selection
+
+`reconcile_retained_project` compares fresh source IDs, keys, types and versions
+with task membership, source relations, project originals and attachment hashes.
+Keep its missing counts and discrepancy list alongside the import report. A
+successful result proves the checked retention scope; it does not itself prove
+the project is ready for cutover.
+
+Run cross-project reconciliation in the operator's normal organisation context
+so accessible organisational relation targets are included alongside private
+projects. A narrower context can report a target as unavailable without proving
+that it is absent.
+
+Database backups do not include separately stored file bytes. Pair the encrypted
+database backup with `scripts/backup_task_project_files.py`: export file copies
+against the restored snapshot, then restore those bytes into an isolated blob
+directory beside the restored database. The manifest binds the database receipt,
+projects, tasks, file identities, sizes and hashes. File restore refuses the
+source database and verifies every restored file through canonical reads. Check
+that representative restored tasks can still retrieve their original content
+and attachments without Jira.
+
+Take the final paired snapshot after migration writes have settled. A database
+dump spanning an in-progress file-copy creation can contain its concept before
+its locator relations; a successful document restore alone does not establish
+a usable content restore.
+
+`set_task_project_writer` records a reversible, per-project writer decision and
+its evidence. Moving to `von` requires successful project reconciliation plus
+site-retention, normal-use, restore and final-delta receipts. The smallest
+delivery evidence includes ordinary browser/MCP operations for the affected
+project, no unexplained missing issues or broken source relations, retrievable
+original content, and a verified final delta. Missing source bytes, incorrect
+visibility, lost native edits, duplicate identity, or an unverified recovery
+path block the corresponding cutover claim. Merely wanting broader analytics,
+typed sprint modelling or Jira UI parity does not.
+
+Until that decision is recorded, Jira remains the writer. Once it is `von`,
+source importers refuse to overwrite or create task projections in that project;
+coding agents use native task tracking and preserve Jira keys as aliases. Jira
+access and retained archives remain available. Do not cancel Jira, delete its
+data, or silently change collaborators' workflows as part of the writer switch.
+
+## Historical implementation slices
+
+The following records describe earlier bounded deliveries. Their completion
+does not certify the current all-project migration.
 
 ## Scope
 

scripts/backup_task_project_files.py

@@ -0,0 +1,226 @@
+"""Back up retained task/project files, or restore them beside a restored database.
+
+Complements restore_von_db.py: MongoDB contains file-copy identities, while the
+original bytes live in blob storage. No Jira connection is needed for either
+operation. Run export against the restored snapshot to bind its exact file set.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+
+def file_copy_ids(value):
+    ids = set()
+    if isinstance(value, dict):
+        for key, item in value.items():
+            if key == "file_copy_concept_id" and isinstance(item, str):
+                ids.add(item)
+            else:
+                ids.update(file_copy_ids(item))
+    elif isinstance(value, list):
+        for item in value:
+            ids.update(file_copy_ids(item))
+    return ids
+
+
+def export_files(*, project_ids, destination, actor_id, database_receipt, cipher):
+    from src.backend.db.mongo_client import get_configured_database_name
+    from src.backend.db.repositories.concepts_repository import ConceptsRepository
+    from src.backend.services.computer_file_copy_service import (
+        fetch_file_copy_bytes,
+        resolve_file_copy_blob_info,
+    )
+    from src.backend.services.task_project_service import (
+        get_task_project,
+        list_project_collections,
+    )
+
+    destination.mkdir(parents=True, exist_ok=True, mode=0o700)
+    identifiers = set()
+    for project_id in project_ids:
+        project = get_task_project(project_id)
+        identifiers.update(file_copy_ids(project))
+        identifiers.update(file_copy_ids(list_project_collections(project)))
+    tasks = list(
+        ConceptsRepository.find(
+            {"metadata.project_concept_id": {"$in": project_ids}},
+            projection={"concept_id": 1, "metadata": 1},
+            limit=0,
+        )
+    )
+    identifiers.update(file_copy_ids(tasks))
+    manifest = {
+        "schema": "task_project_file_backup.v1",
+        "source_database": database_receipt["db_name"],
+        "export_database": get_configured_database_name(),
+        "database_receipt": database_receipt,
+        "project_concept_ids": project_ids,
+        "task_count": len(tasks),
+        "files": [],
+        "complete": False,
+    }
+    for identifier in sorted(identifiers):
+        result = fetch_file_copy_bytes(
+            file_copy_concept_id=identifier, user_concept_id=actor_id, allow_large=True
+        )
+        if not result.get("success"):
+            raise ValueError(f"Cannot back up referenced file copy: {identifier}")
+        data = result["data"]
+        digest = hashlib.sha256(data).hexdigest()
+        encrypted_path = destination / f"{digest}.bin.enc"
+        encrypted_path.write_bytes(cipher.encrypt(data))
+        os.chmod(encrypted_path, 0o600)
+        if (
+            hashlib.sha256(cipher.decrypt(encrypted_path.read_bytes())).hexdigest()
+            != digest
+        ):
+            raise ValueError("Encrypted backup read-back failed")
+        info = resolve_file_copy_blob_info(file_copy_concept_id=identifier)
+        manifest["files"].append(
+            {
+                "file_copy_concept_id": identifier,
+                "sha256": digest,
+                "size_bytes": len(data),
+                "original_filename": info.original_filename,
+                "content_type": info.content_type,
+            }
+        )
+        print(
+            json.dumps({"backed_up": identifier, "size_bytes": len(data)}), flush=True
+        )
+    manifest["complete"] = True
+    (destination / "manifest.json").write_text(json.dumps(manifest, indent=2))
+    return manifest
+
+
+def restore_files(*, backup_dir, destination, actor_id, cipher):
+    from src.backend.db.mongo_client import get_configured_database_name
+    from src.backend.services.blob_store import LocalBlobStore
+    from src.backend.services.computer_file_copy_service import fetch_file_copy_bytes
+    from src.backend.services.concept_service import update_concept
+
+    manifest = json.loads((backup_dir / "manifest.json").read_text())
+    target_db = get_configured_database_name()
+    if target_db in {"von_db", manifest["source_database"]}:
+        raise ValueError("File restore requires a separate restored database")
+    if not manifest.get("complete"):
+        raise ValueError("File backup is incomplete")
+    destination.mkdir(parents=True, exist_ok=True, mode=0o700)
+    store = LocalBlobStore(destination)
+    os.environ["VON_BLOB_STORE_LOCAL_ROOT"] = str(destination.resolve())
+    os.environ["VON_BLOB_STORE_BACKEND"] = "local"
+    restored = []
+    for entry in manifest["files"]:
+        digest = entry["sha256"]
+        if len(digest) != 64 or any(ch not in "0123456789abcdef" for ch in digest):
+            raise ValueError("Invalid file digest")
+        data = cipher.decrypt((backup_dir / f"{digest}.bin.enc").read_bytes())
+        if (
+            len(data) != entry["size_bytes"]
+            or hashlib.sha256(data).hexdigest() != digest
+        ):
+            raise ValueError("File backup integrity mismatch")
+        ref = store.put_bytes(
+            f"restored-task-files/{digest}", data, content_type=entry["content_type"]
+        )
+        identifier = entry["file_copy_concept_id"]
+        update_concept(
+            identifier,
+            {
+                "attributes.file_copy_metadata_storage": "attributes.v1",
+                "attributes.blob_backend": "local",
+                "attributes.blob_key": ref.key,
+                "attributes.blob_uri": ref.uri,
+                "attributes.size_bytes": len(data),
+                "attributes.original_filename": entry["original_filename"],
+                "attributes.content_type": entry["content_type"],
+            },
+        )
+        result = fetch_file_copy_bytes(
+            file_copy_concept_id=identifier, user_concept_id=actor_id, allow_large=True
+        )
+        if (
+            not result.get("success")
+            or hashlib.sha256(result["data"]).hexdigest() != digest
+        ):
+            raise ValueError(f"Restored file canonical read-back failed: {identifier}")
+        restored.append(identifier)
+    receipt = {
+        "schema": "task_project_file_restore.v1",
+        "target_database": target_db,
+        "project_concept_ids": manifest["project_concept_ids"],
+        "task_count": manifest["task_count"],
+        "restored_file_copy_ids": restored,
+        "complete": True,
+    }
+    (destination / "restore-receipt.json").write_text(json.dumps(receipt, indent=2))
+    return receipt
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("operation", choices=["export", "restore"])
+    parser.add_argument("--actor-concept-id", required=True)
+    parser.add_argument("--organisation-concept-id")
+    parser.add_argument("--project-concept-id", action="append", default=[])
+    parser.add_argument("--database-receipt", type=Path)
+    parser.add_argument("--backup-dir", required=True, type=Path)
+    parser.add_argument("--restore-dir", type=Path)
+    args = parser.parse_args()
+    from cryptography.fernet import Fernet
+
+    from src.backend.security.access_control import (
+        force_access_control_enforcement,
+        override_current_actor,
+    )
+
+    cipher = Fernet(os.environ["VON_BACKUP_ENCRYPTION_KEY"].encode())
+    with (
+        override_current_actor(args.actor_concept_id, args.organisation_concept_id),
+        force_access_control_enforcement(),
+    ):
+        if args.operation == "export":
+            if not args.project_concept_id or not args.database_receipt:
+                parser.error(
+                    "Export requires project IDs and the database backup receipt"
+                )
+            result = export_files(
+                project_ids=args.project_concept_id,
+                destination=args.backup_dir,
+                actor_id=args.actor_concept_id,
+                cipher=cipher,
+                database_receipt=json.loads(args.database_receipt.read_text()),
+            )
+        else:
+            if not args.restore_dir:
+                parser.error("Restore requires a separate destination directory")
+            result = restore_files(
+                backup_dir=args.backup_dir,
+                destination=args.restore_dir,
+                actor_id=args.actor_concept_id,
+                cipher=cipher,
+            )
+    print(
+        json.dumps(
+            {
+                "complete": result["complete"],
+                "file_count": len(
+                    result.get("files", result.get("restored_file_copy_ids", []))
+                ),
+            }
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

scripts/backup_von_db.py

@@ -522,7 +522,8 @@ def main(argv: list[str] | None = None) -> int:
     _ensure_dir(backup_root)
 
     # Write a prelude file at the backup root (helpful provenance when copying around).
-    (backup_root / "prelude.json").write_text(
+    # prelude.json is reserved by MongoDB Database Tools for server metadata.
+    (backup_root / "von_backup_prelude.json").write_text(
         json.dumps(asdict(prelude), indent=2, ensure_ascii=False) + "\n",
         encoding="utf-8",
     )