Skip to content

README.md

Latest commit

 

History

History
137 lines (106 loc) · 7.34 KB

README.md

File metadata and controls

137 lines (106 loc) · 7.34 KB
Raw

Reactome status page

Static status page for Reactome production services, published at https://status.reactome.org. Each production host runs a small collector every 5 minutes that uploads a JSON snapshot to S3; CloudFront serves the bucket. Because the page is static and the data lives in S3, it stays up when a production server does not, and a missing report is the outage signal.

Alpha. Not yet an official Reactome service; numbers may be incomplete or wrong.

Layout

Path What
collector/collector.py Stdlib-only Python 3 collector. systemd units, HTTP/TCP health checks, Apache mod_status, access-log aggregates, host load/memory/disk, restart detection. Uploads with the AWS CLI. Containers are checked over HTTP/TCP; the collector deliberately has no Docker socket access.
collector/config/<host>.json Per-host configuration (what to check, log path, S3 prefix).
collector/*.service, *.timer, install.sh systemd units and the sudo install script.
site/ The static page: index.html, app.js, style.css, vendored uPlot, hosts.json (list of hosts shown).
infra/status-site.yaml CloudFormation: private S3 bucket, CloudFront (OAC, HTTPS), lifecycle rule, IAM upload policy.
infra/deploy.sh Certificate request, stack deploy, site upload.
docs/PLAN.md Original design notes.
docs/BACKLOG.md Pointer to the backlog, which lives in GitHub issues.

Data in the bucket

data/<host>/latest.json        current snapshot            (Cache-Control 60 s)
data/<host>/series/24h.json    5-min points, last 24 h
data/<host>/series/7d.json     30-min points, last 7 d
data/<host>/series/90d.json    6-h points, last 90 d
data/<host>/events.json        the 500 most recent restart / outage / recovery events
raw/<host>/YYYY/MM/DD/HHMM.json  every snapshot, expired after 90 d by S3 lifecycle
site/                          the page itself (served via a CloudFront origin path; hosts cannot write here)

Only aggregates leave a host: request counts, status-class counts and latency percentiles per URL group. No client IPs, user agents or full URLs are uploaded. The page is public, so nothing in a snapshot should be sensitive.

Using the page

  • The 24 h / 7 d / 90 d buttons set the range for the whole page; the choice is remembered per browser.
  • Click the ⤢ icon on any chart (or double-click the plot) to enlarge it. The enlarged view has its own range control plus a custom From/To window; it uses the finest history that covers the window and says which resolution it is showing. "Download CSV" exports the shown window; "Copy link" copies a URL that reopens exactly that chart and window (the address bar always reflects the current view).
  • The ◐ button cycles the colour theme: follow the system, light, dark.

Deploying the site (once)

infra/deploy.sh cert     # request ACM cert; add the printed CNAME in Cloudflare (DNS only)
infra/deploy.sh stack    # once the cert is ISSUED: bucket + CloudFront + IAM policy
infra/deploy.sh outputs  # DistributionDomainName -> Cloudflare CNAME for status.reactome.org (DNS only)
infra/deploy.sh site     # upload site/ (re-run after any page change)

The stack attaches an upload policy to the production hosts' existing instance role (EC2CloudwatchAgentRole), so the collector needs no access keys.

Installing the collector on a host

git clone https://github.com/reactome/reactome-status.git
sudo reactome-status/collector/install.sh reactome-status/collector/config/reactome.org.json

This creates a reactome-status system user with its own group, grants it read access to the Apache log directory with a narrow ACL (not group membership, which would also expose other group-readable files such as CMS credentials; deliberately not docker either, which would be root-equivalent), installs the script under /opt/reactome-status, the config under /etc/reactome-status, state under /var/lib/reactome-status, and enables a timer that runs at minutes 1, 6, 11, … past the hour (deliberately off the :00/:30 marks that cron jobs use). Check with:

systemctl list-timers reactome-status-collector.timer
journalctl -u reactome-status-collector.service -n 20

Stop it with sudo systemctl disable --now reactome-status-collector.timer, or remove it entirely with sudo collector/uninstall.sh. The unit is capped at half a CPU and 512 MB and runs with a read-only view of the filesystem.

To try it without uploading (any user in the reactome group), point it at a private state directory so it does not touch the installed collector's history:

python3 collector/collector.py -c collector/config/reactome.org.json --state-dir /tmp/status-test --no-upload --print

Adding another host (e.g. curator.reactome.org)

  1. Copy collector/config/reactome.org.json to collector/config/curator.reactome.org.json, set host, s3_prefix (data/curator.reactome.org), and adjust units, probes and log path.
  2. Add the host to site/hosts.json and run infra/deploy.sh site.
  3. Make sure the host's instance role has the upload policy (stack output CollectorUploadPolicyArn).
  4. Run install.sh on the host.

Spec-driven development

The repo uses Spec Kit. .specify/memory/constitution.md records the project's principles (independence from what it monitors, missing data is downtime, only aggregates leave a host, least privilege, inert uploads, …); every non-trivial change is checked against it. Features live under specs/:

  • specs/000-status-page-baseline/ describes the system as delivered, for /speckit-converge.
  • specs/001-multi-host-monitoring/ is the next feature (curator, Plant Reactome, CPWS): spec, plan, research, data model, operator quickstart and task list.

Workflow in Claude Code: /speckit-specify (new feature) → /speckit-clarify (optional) → /speckit-plan → /speckit-tasks → /speckit-implement; /speckit-analyze checks the artifacts against each other and the constitution; /speckit-converge compares the codebase with a spec and appends remaining work as tasks. The commands act on the feature named in the machine-local .specify/feature.json (not committed); set SPECIFY_FEATURE_DIRECTORY or edit that file to point at another feature, e.g. specs/000-status-page-baseline. The constitution says which changes need a spec; small fixes skip the ceremony.

Collector self-test

collector/selftest.sh runs the collector against a fake access log in a scratch directory and checks first-run behaviour, grouping, rotation carry-over, the unreadable-log path and the root/unsafe-run refusals. Run it before committing collector changes; the constitution's "verified before deployed" principle relies on it together with a --state-dir dry run on a host.

Local development of the page

# pull real snapshots from a host into site/data/ (git-ignored) then serve
cd site && python3 -m http.server 8765

Restart tracking

A restart is detected when a service's start time (systemd ExecMainStartTimestamp or Docker StartedAt) changes between runs. The collector then records when the service's health check first passes again, giving a time-to-healthy with 5-minute resolution. Events appear in events.json and in the "Recent service events" table.