Repository navigation
Storage: Standardize S3 paths #1170
Description
Is your feature request related to a problem?
Different modules are inconsistent in how they store objects in S3, which complicates data management and raises security concerns.
Describe the solution you'd like
- Standardize all storage paths to use
project.storage_pathfor documents, voice notes, eval datasets, batch data, and LLM call/chain artifacts. - Remove the per-module cleanup logic to facilitate project-reference-based deletion of data from S3.
Original issue
Context
Different modules store objects in S3 differently — documents use project.storage_path (a UUID-based path, good), while audio/TTS-STT paths embed org/project IDs directly.
Decision
Everything (documents, voice notes, eval datasets, batch data, LLM call/chain artifacts) must use project.storage_path.
Rationale
- Security: UUID paths mean S3 access alone can't be correlated to an org/project without DB access.
- Cleanup: enables deleting all of a project's data from S3 by project reference (DPDP-relevant), and removes per-module if/else cleanup logic.
Acceptance criteria
- Documents, LLM call, LLM chain, and eval artifacts all resolve their S3 paths via
project.storage_path. - Per-module cleanup if/else logic is removed in favor of project-reference-based deletion.
Activity
[-]Standardize S3 storage paths to project.storage_path across documents, LLM call/chain, and evals[/-][+]Storage: Standardize S3 paths[/+]