Skip to content

devops.md

Latest commit

 

History

History
244 lines (178 loc) · 9.59 KB

devops.md

File metadata and controls

244 lines (178 loc) · 9.59 KB
Raw

veep — DevOps

Internal-only. Never propagated to github.com/vectorpanda/veep. The mirror procedure below excludes this file via rsync --exclude='devops.md'. If you add new internal-only files in veep/, extend that exclude list.

Where things live

What Where
Source of truth for veep/ This monorepo (Forgejo at 10.1.1.8)
Public mirror github.com/vectorpanda/veep — 1 commit per release
Published artifacts pypi.org/project/veep + test.pypi.org/project/veep
Default host customers hit https://api.vectorpanda.com → rampfee → consumer-site .100
TLS cert lineage /etc/letsencrypt/live/vectorpanda.com/ on rampfee.me
~/.pypirc (manual twine creds) /home/mike/.pypirc on .200, perms 0600

Accounts

  • pypi.org: account owned by vectorpandasite@gmail.com (same Google identity that owns the consumer-site OAuth client per the auto-memory).
  • test.pypi.org: same account.
  • github.com/vectorpanda: organization. The veep repo lives there as a public mirror.
  • PyPI / TestPyPI API tokens: minted via web UI. Tokens are shown once at creation; if lost, revoke and re-mint.
    • Token shape: pypi-AgE[IN]...; project-scoped tokens carry an extra caveat block (~30 chars longer than account-scoped). Test/prod tokens use the same prefix and structure — the embedded location header (pypi.org vs test.pypi.org) is the only intrinsic difference.

Publishing flow

CI uses PyPI Trusted Publishing (OIDC) — there is no PyPI API token stored in GitHub secrets. The handshake is between the GitHub Actions runner's OIDC token and PyPI's "Trusted Publisher" config. If publish ever 403s, that config is always the first thing to check.

Publish a new release (procedure)

# 1. Bump version — pyproject.toml is the ONLY place. src/veep/__init__.py
#    derives __version__ from installed package metadata at import time
#    (importlib.metadata), so the historical "bump both in lockstep" step
#    (server-k6sq era) is gone: a sed against a literal __version__ string
#    in __init__.py silently matches nothing (observed 2026-09-07 on the
#    0.5.33 bump). test_version.py still guards the derived value.
cd /home/mike/server/veep
OLD=0.5.32 NEW=0.5.33   # edit these
sed -i "s/version = \"$OLD\"/version = \"$NEW\"/" pyproject.toml
grep -E "^version" pyproject.toml   # eyeball it moved

# 2. Validate locally
.venv/bin/pytest -q                              # must be 100% green
rm -rf dist/ build/ src/veep.egg-info/
.venv/bin/python -m build
.venv/bin/twine check dist/*

# 3. (Recommended) TestPyPI dry-run — catches metadata bugs before they're permanent
.venv/bin/twine upload --repository testpypi dist/*
# eyeball https://test.pypi.org/project/veep/X.Y.Z/
# install in a throwaway venv:
python3 -m venv /tmp/veep-test && /tmp/veep-test/bin/pip install \
  --index-url https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple/ \
  veep==X.Y.Z

# 4. Mirror to GitHub
rm -rf /tmp/veep-mirror && mkdir -p /tmp/veep-mirror
rsync -a \
  --exclude='.venv/' \
  --exclude='dist/' \
  --exclude='build/' \
  --exclude='__pycache__/' \
  --exclude='*.pyc' \
  --exclude='.pytest_cache/' \
  --exclude='.ruff_cache/' \
  --exclude='*.egg-info/' \
  --exclude='devops.md' \
  --exclude='CLAUDE.md' \
  --exclude='API_QUESTIONS.md' \
  /home/mike/server/veep/ /tmp/veep-mirror/

cd /tmp/veep-mirror
git init -q -b main
git add .
git -c user.email=mike@vectorpanda.com -c user.name=mike commit -q -m "Release X.Y.Z"
git remote add origin git@github.com:vectorpanda/veep.git
git tag vX.Y.Z

# 5. Push — tag push triggers publish.yml on GitHub Actions
git push -f origin main          # -f because each release is a fresh tree (no shared history)
git push origin vX.Y.Z

# 6. Watch the workflow (~30s end-to-end)
curl -sS "https://api.github.com/repos/vectorpanda/veep/actions/runs?per_page=1" \
  | python3 -c "import json,sys; r=json.load(sys.stdin)['workflow_runs'][0]; print(r['status'], r['conclusion'] or '—', r['html_url'])"

# 7. Verify on real PyPI
curl -sS https://pypi.org/pypi/veep/json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['info']['version'])"
# Smoke install:
python3 -m venv /tmp/veep-real && /tmp/veep-real/bin/pip install veep==X.Y.Z
/tmp/veep-real/bin/python3 -c "from veep import VP; print(VP(api_key='x').ping())"

Trusted Publisher entries

Both PyPI and TestPyPI need a Trusted Publisher entry pointing at the same workflow:

Field Value
PyPI Project Name veep
Owner vectorpanda
Repository veep
Workflow filename publish.yml
Environment pypi

URLs:

The environment name pypi matches the environment: pypi line in .github/workflows/publish.yml. If you change one, change the other.

SSH deploy key (mirror push)

The monorepo host (.200, this shell) has an SSH key registered as a deploy key with write access on github.com/vectorpanda/veep. Pubkey:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPSGlt0Tm5Jxd140DW3ywg5BjECsDHyJpOaqbxoCST8F mike@dev-vm

Without it, git push to the mirror fails with Permission denied (publickey).

~/.pypirc

/home/mike/.pypirc (perms 0600):

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
  username = __token__
  password = pypi-...

[testpypi]
  repository = https://test.pypi.org/legacy/
  username = __token__
  password = pypi-...

These tokens are only used for manual twine runs — primarily TestPyPI dry-runs before a release. The real-PyPI release flow uses Trusted Publishing and never reads this file.

If a token is lost or compromised: revoke at the PyPI/TestPyPI web UI, mint a new one, replace the value in ~/.pypirc directly. Never paste tokens into chat or commit them.

TLS / api.vectorpanda.com

veep's DEFAULT_HOST is https://api.vectorpanda.com (set in veep/src/veep/client.py). Customers' very first network call goes there.

  • DNS: api.vectorpanda.com → 50.117.121.82 (rampfee.me)
  • Cert lineage: /etc/letsencrypt/live/vectorpanda.com/ on rampfee
  • SANs: vectorpanda.com, www.vectorpanda.com, api.vectorpanda.com
  • Renewal: certbot cron (default schedule)

To add another SAN later (e.g. a new sub-domain customers should reach):

ssh root@10.1.1.240 'certbot --nginx --expand --cert-name vectorpanda.com \
  -d vectorpanda.com -d www.vectorpanda.com -d api.vectorpanda.com \
  -d <new-name> --non-interactive'

--expand is the load-bearing flag — it operates on the existing lineage instead of creating a new one. Without it certbot creates a parallel lineage and renewal cron tries to renew both, which gets messy.

Signup brake interplay

Customers can pip install veep regardless of brake state, but actually creating an account routes through the consumer-site's /signup page, which honors the brake.

  • Brake ON (signup_enabled=false, current state on prod .101): self-service signups go to the waitlist; existing API keys keep working.
  • Brake OFF (signup_enabled=true): full self-service via Google/GitHub OAuth.

Toggle:

curl -X POST -H "X-Admin-Auth: $PROD_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"enabled": true}' \
  http://10.1.1.101:3000/admin/signup_enabled

When you flip the brake OFF (general availability), also remove the "Beta Status" paragraph from the README and bump a version (Stale-paths principle: the README claims private beta; if signups are open, it's not true).

Yank / rollback

PyPI versions are immutable — there's no "delete and re-upload."

If a published version has a bug:

  1. Yank at https://pypi.org/manage/project/veep/release/X.Y.Z/ → "Yank release". Yanked versions stay installable for existing pinners but are skipped by pip install veep without a pin. Yank is reversible.
  2. Bump pyproject.toml to the next patch (e.g. 0.4.2 → 0.4.3).
  3. Fix the bug.
  4. Re-run the publishing procedure.

Never try to overwrite a version. PyPI's 403 on duplicate-filename uploads is by design.

Per-bead history

  • server-gzb.3.1 — Publish veep SDK to PyPI. Closed 2026-05-06.
  • server-gzb.3.3 — Create github.com/vectorpanda/veep public repo. Closed 2026-05-06.
  • server-3t7 — 5-step quickstart README. Closed 2026-05-06.
  • server-rtar — api.vectorpanda.com TLS SAN. Closed 2026-05-06.
  • server-2upn — rampfee.me maintenance backlog (CentOS 7 EOL, old certbot/nginx, conf cleanup). Open, P3.
  • server-gzb.3.2 — CSV / binary / Safetensors format support in upsert(). Open, P1. README's Upload Modes section currently lists only what works; expand when 3.2 lands.

Quick reference (the things you actually need at 2 AM)

# What version is published?
curl -sS https://pypi.org/pypi/veep/json | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["version"])'

# Is the customer-facing TLS healthy?
curl -sS -o /dev/null -w "%{http_code}\n" https://api.vectorpanda.com/api/v1/health

# What's the cert SAN list right now?
echo | openssl s_client -connect api.vectorpanda.com:443 -servername api.vectorpanda.com 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName -dates 2>/dev/null

# Last GH Actions publish run
curl -sS "https://api.github.com/repos/vectorpanda/veep/actions/runs?per_page=1" \
  | python3 -c "import json,sys; r=json.load(sys.stdin)['workflow_runs'][0]; print(r['head_branch'], r['status'], r['conclusion'], r['html_url'])"

# Brake state on prod
curl -sS -H "X-Admin-Auth: $PROD_ADMIN_TOKEN" http://10.1.1.101:3000/admin/signup_enabled