Streaming group export is a non-atomic multi-query read: concurrent ingest mid-stream yields an internally inconsistent aggregate (events referencing sources not emitted)

[erskingardner]

Summary

api_group_export_stream builds several sections (source audit files, events, delivery_artifacts, …) that stream_ndjson iterates and emits sequentially, each as its own queryset evaluation, after the view has returned. No transaction.atomic/snapshot spans the sections, so an upload committed to the same group between two section queries produces an export that is internally inconsistent, despite the docstring promising a "complete self-contained aggregate."

Location

forensics/views.py:3228-3284 (the sections list streamed via stream_ndjson; no snapshot spans the body).

Failure scenario

Uploads are frequent in this tool. The source section (audit files) is fully streamed before the event/delivery_artifact sections are queried. If another upload to the same group commits in that window:

• The later event/delivery_artifact rows include events whose audit_file_id points at a source file never emitted in the already-finished source section.
• The reverse interleaving can drop just-added events instead.

The documented consumer (the CGKA pipeline — "a complete self-contained download", "every fork_resolution must be present") then receives events referencing absent sources: an internally inconsistent export. Because the stream still terminates with a normal eof line, the consumer's "last line is not eof → incomplete" guard does not catch this torn read.

Notes / mitigation

A StreamingHttpResponse cannot trivially hold one DB transaction across the whole response body (Django's ATOMIC_REQUESTS does not wrap streaming responses), so this is partly inherent. Options:

• Wrap the queryset evaluation in a REPEATABLE READ snapshot for the duration of the stream (server-side cursors already in use make this feasible on Postgres), or
• Emit a monotonic watermark (e.g. max event id / upload timestamp) in the manifest and filter every section to <= watermark, so the export is a consistent point-in-time cut, or
• At minimum, document the torn-read window and the consumer's exposure.

Relationship to existing issues

#219 (raw_text per-chunk memory), #295 (sensitivity under-declaration), and #262 (flag-vs-auth ordering) concern the streaming export but none address transactional/snapshot consistency of the stream.

Filed by an automated code-review pass.