cold-archive-claw

A small background job for macOS that moves old video files off your laptop into a Google Cloud Storage bucket using the Archive storage class, and keeps an index that you, or an LLM agent, can search to find and restore anything.

At about $0.0012 per GB per month, a terabyte of old footage costs roughly $1.20 a month to keep.

How it works

Every few hours (6 by default), launchd runs one pass:

1. Scan the folders in ARCHIVE_ROOTS for video files (.mp4 .mov .m4v .mkv .webm …) that nobody has modified for ARCHIVE_MIN_AGE_DAYS days (default 60). The scan skips:
   • hidden folders
   • node_modules and .git
   • app and library bundles (.photoslibrary, .fcpbundle, .app, …), so libraries stay intact
   • anything matching ARCHIVE_EXCLUDE
2. Hash each file locally (MD5).
3. Upload it to gs://$ARCHIVE_BUCKET/videos/<machine>/<path under your home>. The bucket's default class is Archive.
4. Verify: read the object's MD5 back from GCS. If it doesn't match, the pass logs an error, keeps the local file and records nothing.
5. Record a manifest entry with:
   • original path, bucket URI and size
   • MD5
   • modified and archived dates
   • length, resolution, codec and audio (from ffprobe)
6. Free disk (optional). With ARCHIVE_DELETE_LOCAL=true, the local file is deleted only when the upload verified and the file's size and mtime are unchanged since it was hashed. A file you touched mid-upload is never removed.
7. Publish the index to the bucket every 25 files during a pass and again at the end, in the Standard storage class so reading it is free of retrieval fees:
   • index/manifest.jsonl: one JSON object per file (the source of truth)
   • index/INDEX.md: the same data as Markdown tables, grouped by the folder each file came from

Other guarantees:

• Fresh files are skipped. Anything modified within the age window is left alone, so work in progress is never touched.
• Already-archived files are skipped on later passes.
• Passes never overlap. A lock file prevents it.
• One failure doesn't stop the pass. The error is logged and the next file is tried.

Archive-class objects stay online: restoring takes seconds, not hours. The trade-offs are a retrieval fee (about $0.05/GB) and a 365-day minimum storage charge per object, so this suits files you rarely need back.

Setup

Requirements: macOS, Python 3.9+, the gcloud CLI signed in (gcloud auth login), and optionally ffprobe (brew install ffmpeg) for video metadata. There are no Python dependencies.

# 1. a private Archive-class bucket
gcloud storage buckets create gs://my-cold-archive --location=us-central1 \
  --default-storage-class=ARCHIVE --uniform-bucket-level-access --public-access-prevention

# 2. config
git clone https://github.com/Varritech/cold-archive-claw && cd cold-archive-claw
./install.sh                       # creates ~/.cold-archive/.env from .env.example
$EDITOR ~/.cold-archive/.env       # set ARCHIVE_BUCKET, roots, delete policy

# 3. preview, then install the background job
bin/cold-archive run --dry-run
./install.sh

macOS privacy: launchd jobs can't read Desktop, Documents or Downloads until you give the Python binary Full Disk Access (System Settings → Privacy & Security → Full Disk Access → +, then pick the path install.sh prints). Without it, the log says exactly that and the pass exits.

Configuration

All settings are environment variables, read from ~/.cold-archive/.env (or from $COLD_ARCHIVE_ENV).

Variable	Default	Meaning
ARCHIVE_BUCKET	(required)	Target bucket name
ARCHIVE_ROOTS	~/Downloads:~/Desktop:~/Movies	Colon-separated folders to scan
ARCHIVE_MIN_AGE_DAYS	60	Only files unmodified this long
ARCHIVE_DELETE_LOCAL	false	Delete local copy after a verified upload
ARCHIVE_PREFIX	hostname	Folder under videos/ for this machine
ARCHIVE_EXCLUDE	none	Comma-separated path globs to never archive
ARCHIVE_EXTENSIONS	common video types	Comma-separated extensions
ARCHIVE_MAX_GB_PER_RUN	0 (no limit)	Cap per pass
ARCHIVE_STATE_DIR	~/.cold-archive	Manifest, INDEX.md, logs, lock
ARCHIVE_INTERVAL_HOURS	6	Read by install.sh only

Finding and restoring files

bin/cold-archive search webinar sophia     # every term must appear in the original path
bin/cold-archive search launch --json      # machine-readable
bin/cold-archive restore "promo v2"        # back to its original path
bin/cold-archive restore promo --to ~/tmp --yes
bin/cold-archive status
bin/cold-archive sync-index                # on another machine: pull the bucket's manifest locally

For LLM agents

The index is plain text in the bucket, so an agent with gcloud access needs no custom tooling:

gcloud storage cat gs://$BUCKET/index/INDEX.md          # overview, grouped by folder
gcloud storage cat gs://$BUCKET/index/manifest.jsonl | grep -i "keyword"
gcloud storage cp "gs://$BUCKET/videos/<machine>/<path>" ./   # fetch one file

Each manifest line looks like this:

{"key": "videos/studio-mac/Desktop/renders/promo v2.mp4", "uri": "gs://my-cold-archive/videos/studio-mac/Desktop/renders/promo v2.mp4",
 "original_path": "/Users/me/Desktop/renders/promo v2.mp4", "filename": "promo v2.mp4", "size": 48213300,
 "md5": "…", "modified": "2026-05-02T14:11:09+00:00", "archived_at": "2026-07-01T03:00:12+00:00",
 "duration_s": 42.1, "width": 1080, "height": 1920, "codec": "h264", "has_audio": true,
 "host": "studio-mac", "local_deleted": true}

Tests

python3 -m unittest discover -s tests

The tests cover the scan rules, the pass (upload, skip, dry run, index publishing), every delete-safety rule, search and config. GCS is swapped for an in-memory fake at the storage boundary.

Uninstall

./install.sh uninstall removes the launchd job. The bucket and the index stay as they are.

License

MIT