diff --git a/AGENTS.md b/AGENTS.md index 60f6f78..cb664ab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,6 +55,7 @@ The sample-workspace seed is not the public demo. `/demo` is component-memory-on - Standalone projects and membership contracts/validation: `src/lib/projects/`; project HTTP routes: `src/app/api/projects/`, `src/app/api/studies/[id]/project/` - Project storage: `src/lib/storage/redisProjects.ts`, `cloudflare/workspace/projects.ts`, with the project sub-port in `src/lib/storage/types.ts` and RPC validation in `src/lib/storage/durableObject.ts` +- Study ZIP analysis views: `src/lib/export/analysisFiles.ts`; both ZIP paths use these snapshot-only builders. Combined entries stream from repeated reads of the same fenced snapshot; turn CSVs are per-interview shards. Keep citation indices 1-based over the full stored transcript, including omitted system-message positions. - Shared per-study transcript sources and bounded project composition: `src/lib/export/studyTranscriptsSource.ts`, `src/lib/export/projectTranscriptsMarkdown.ts`; typed browser client: `src/services/projectService.ts` - Researcher studies/interviews and atomic Redis scripts: `src/lib/kv.ts` - Field-level JSON patching inside Redis Lua (preserves untouched value types): `src/lib/studyJsonLua.ts` diff --git a/README.md b/README.md index 302f293..7919af9 100644 --- a/README.md +++ b/README.md @@ -331,7 +331,19 @@ For researchers: 6. Choose a dataset, run aggregate analysis, or ask questions in **Explore**; export the study or workspace. 7. Pause and resume collection without replacing links, or delete a study in Settings when its retention period ends. -**Export this study** downloads a ZIP of raw records, transcripts, the aggregate and the notebook. **Export transcripts (.md)** downloads one Markdown file with every saved transcript of the study, for reading or for another analysis tool. Transcript text is quoted exactly as saved, and each interview lists what its participant was told about the AI. If a participant was promised that the study uses only one provider and model, sending the file to a different AI service may break that promise. A download that did not finish is refused rather than saved. +**Export this study** downloads a ZIP of raw records, transcripts, the aggregate and the notebook, plus an `analysis/` folder for coding tools and spreadsheets. The same analysis files are included in workspace ZIP exports. + +The ZIP retains the per-interview `NNN_*.json` and `.md`, `summary.csv`, `aggregates/` and `explorations/`. It adds: + +- `analysis/README.md`: field definitions, historical-label rules and the consent/provider sharing warning. +- `analysis/interviews.jsonl`: one interview per line, with collection context, recorded consent and analysis provenance, profile values and verbatim turns. +- `analysis/interviews.csv`: interview-level data and one column per observed profile field ID, labeled from the newest interview. +- `analysis/turns/NNN.csv`: one turn table per interview to bound export memory; indices match analysis citations, including gaps for omitted system messages. +- `analysis/profile_fields.csv`: profile values in long form with their labels at collection. + +CSVs use UTF-8 with BOM and CRLF. Formula-like text is protected with an added apostrophe; use JSONL for verbatim quotes. Unknown historical fields are not filled from today’s study configuration. ZIP Markdown dates are ISO UTC. + +**Export transcripts (.md)** downloads one Markdown file with every saved transcript of the study, for reading or for another analysis tool. Transcript text is quoted exactly as saved, and each interview lists what its participant was told about the AI. If a participant was promised that the study uses only one provider and model, sending the file to a different AI service may break that promise. A download that did not finish is refused rather than saved. ### Explore this study diff --git a/docs/releases/v5.4.0.md b/docs/releases/v5.4.0.md index 427781f..bb7df11 100644 --- a/docs/releases/v5.4.0.md +++ b/docs/releases/v5.4.0.md @@ -75,3 +75,19 @@ browser locality or language-pack support. - Run the workers/Redis shared revision contract and browser lanes listed in `CHANGES-device-voice.md`. No deployment or real browser verification was done in the restricted implementation sandbox. + +## Richer export + +Study and workspace ZIP downloads now include an `analysis/` folder for AI coding tools and spreadsheets. No new export button or third-party adapter is required. + +- `README.md` defines every field and repeats the Markdown export’s consent/provider sharing warning. +- `interviews.jsonl` holds one interview per line with its saved collection protocol, consent and provider records, synthesis, profile and verbatim turns. +- `interviews.csv` offers an interview-level table, including dynamic profile columns labeled from the newest interview. +- `turns/NNN.csv` provides a turn table per interview, keeping export memory bounded. Turn indices match analysis citations: skipped system messages leave gaps. +- `profile_fields.csv` preserves each profile value’s historical label and extraction status. + +CSV files use UTF-8 with BOM and CRLF. Formula-like text receives a protective apostrophe; JSONL is the source for verbatim quotations. Missing historical configuration remains missing, never reconstructed from the current study. Per-interview Markdown timestamps in the ZIP now use ISO UTC. + +Node and Cloudflare produce the same file contents. Cloudflare reads the same captured snapshot in bounded passes, and the archive is finalized only after all files and the final sequence check succeed. Existing raw JSON, Markdown, summary CSV, aggregate and notebook exports remain included. + +Before sharing these files with another AI service, check each participant’s recorded provider commitment. Exporting data does not change that promise. diff --git a/src/app/api/interviews/export/route.ts b/src/app/api/interviews/export/route.ts index 5de5a4d..6e89fbd 100644 --- a/src/app/api/interviews/export/route.ts +++ b/src/app/api/interviews/export/route.ts @@ -27,6 +27,7 @@ import { MAX_OWNED_STUDIES, } from '@/lib/ownedStudies'; import JSZip from 'jszip'; +import { analysisCsvHeader, analysisInterviewsHeader, analysisInterviewsRow, analysisJsonLine, analysisProfileRows, analysisReadme, analysisTurnsCsv, analysisTurnsName, collectProfileColumns, PROFILE_COLUMNS, sortedProfileColumns, type ProfileColumns } from '@/lib/export/analysisFiles'; import { StoredInterview, StoredAggregateSynthesis, type StoredStudy } from '@/types'; import { logRequestFailure } from '@/lib/requestLog'; import { @@ -116,6 +117,8 @@ async function buildExportResponse( beforeReturn?: () => Promise, ): Promise { const zip = new JSZip(); + zip.folder('analysis/turns'); + const columns: ProfileColumns = new Map(); // Entry names and contents come from the builders the Cloudflare stream // uses, so the two archives cannot drift apart. @@ -123,6 +126,8 @@ async function buildExportResponse( const baseName = interviewEntryBaseName(index, interview); zip.file(`${baseName}.json`, interviewJson(interview)); zip.file(`${baseName}.md`, interviewTranscriptMarkdown(interview)); + zip.file(analysisTurnsName(index), analysisTurnsCsv(interview)); + collectProfileColumns(columns, interview); }); for (const [studyId, aggregate] of aggregates) { @@ -134,6 +139,11 @@ async function buildExportResponse( const csvLines = [SUMMARY_CSV_HEADER, ...interviews.map(summaryCsvRow)]; zip.file(SUMMARY_CSV_NAME, csvLines.join('\n')); + const orderedColumns = sortedProfileColumns(columns); + zip.file('analysis/interviews.jsonl', interviews.map(analysisJsonLine).join('')); + zip.file('analysis/interviews.csv', analysisInterviewsHeader(orderedColumns) + interviews.map(interview => analysisInterviewsRow(interview, orderedColumns)).join('')); + zip.file('analysis/profile_fields.csv', analysisCsvHeader(PROFILE_COLUMNS) + interviews.map(analysisProfileRows).join('')); + zip.file('analysis/README.md', analysisReadme(orderedColumns)); const zipBlob = await zip.generateAsync({ type: 'blob' }); const refusal = await beforeReturn?.(); @@ -169,7 +179,7 @@ async function streamDurableExport(store: DurableWorkspaceStorePort, studyId?: s // Every page carries at least one row or aggregate, which bounds the walk. const maximumPages = begun.count + begun.studyIds.length + MAX_EXPLORATION_ANSWERS + 1; const body = createInterviewExportStream({ - pages: remainingExportPages(store, sequence, first, maximumPages, studyId), + pages: () => remainingExportPages(store, sequence, first, maximumPages, studyId), beforeFinish: async () => { const verified = await store.verifyExportSequence({ sequence, ...scope }); if (verified === 'changed') throw new ExportSnapshotChangedError(); diff --git a/src/components/Dashboard.tsx b/src/components/Dashboard.tsx index 1c29183..ffdfcb2 100644 --- a/src/components/Dashboard.tsx +++ b/src/components/Dashboard.tsx @@ -213,6 +213,7 @@ export default function Dashboard() { - + diff --git a/src/lib/export/analysisFiles.ts b/src/lib/export/analysisFiles.ts new file mode 100644 index 0000000..f5d38fd --- /dev/null +++ b/src/lib/export/analysisFiles.ts @@ -0,0 +1,239 @@ +// Portable, snapshot-only analysis views. Never reconstruct history from a live study. +import type { StoredInterview } from '@/types'; +import { analysisStatus } from '@/lib/analysisState'; +import { csvCell } from '@/lib/csv'; +import { LANGUAGE_TAGS } from '@/lib/i18n/languages'; +import { AI_SHARING_WARNING, escapeMarkdownInline } from './transcriptsMarkdown'; + +export const ANALYSIS_SCHEMA = 'openinterviewer.analysis.v1'; +export const DEFINITION_UNAVAILABLE = '(definition unavailable)'; +const iso = (timestamp: number) => new Date(timestamp).toISOString(); + +export function analysisProfile(interview: StoredInterview) { + return { + fields: (interview.participantProfile?.fields ?? []).map(field => ({ + fieldId: field.fieldId, + label: interview.collectionConfig?.profileSchema.find(definition => definition.id === field.fieldId)?.label ?? DEFINITION_UNAVAILABLE, + value: field.value, + status: field.status, + })), + context: interview.participantProfile?.rawContext, + }; +} + +export function analysisTurns(interview: StoredInterview) { + return interview.transcript.flatMap((message, index) => message.role === 'system' ? [] : [{ + index: index + 1, + messageId: message.id, + role: message.role, + speaker: message.role === 'user' ? 'Participant' as const : 'Interviewer' as const, + timestamp: iso(message.timestamp), + offsetSeconds: (message.timestamp - interview.createdAt) / 1000, + text: message.content, + }]); +} + +export function analysisInterview(interview: StoredInterview) { + const config = interview.collectionConfig; + return { + schema: ANALYSIS_SCHEMA, + study: { + id: interview.studyId, name: interview.studyName, revision: interview.studyRevision, + researchQuestion: config?.researchQuestion, coreQuestions: config?.coreQuestions, + topicAreas: config?.topicAreas, interviewLanguages: config?.interviewLanguages, + }, + interview: { + id: interview.id, status: interview.status, + createdAt: iso(interview.createdAt), completedAt: iso(interview.completedAt), + durationMinutes: (interview.completedAt - interview.createdAt) / 60_000, + language: interview.interviewLanguage, + languageTag: interview.interviewLanguage ? LANGUAGE_TAGS[interview.interviewLanguage] : undefined, + voiceInput: config ? config.voiceInput ?? 'off' : undefined, + }, + consent: { + providerCommitment: interview.providerCommitment, + conductedBy: { provider: interview.conductedByProvider, model: interview.conductedByModel }, + transport: interview.consentTransport ?? 'direct', + consentHash: interview.consentHash, + consentAcceptedAt: interview.consentAcceptedAt === undefined ? undefined : iso(interview.consentAcceptedAt), + }, + analysis: { + status: analysisStatus(interview), provider: interview.aiProvider, model: interview.aiModel, + transport: interview.aiTransport ?? 'direct', + bottomLine: interview.synthesis?.bottomLine, themes: interview.synthesis?.themes, + }, + profile: analysisProfile(interview), + turns: analysisTurns(interview), + }; +} + +export function analysisJsonLine(interview: StoredInterview): string { + return `${JSON.stringify(analysisInterview(interview))}\n`; +} + +/** CSV record separators AND embedded text newlines are CRLF; JSONL retains exact text. */ +export function analysisCsvRow(cells: readonly unknown[]): string { + return cells.map(cell => { + // A finite number cannot carry a formula; guarding it would turn -0.5 into text. + if (typeof cell === 'number' && Number.isFinite(cell)) return `"${String(cell)}"`; + return csvCell(cell === undefined || cell === null ? '' : String(cell).replace(/\r\n?|\n/g, '\r\n')); + }).join(',') + '\r\n'; +} +export const analysisCsvHeader = (cells: readonly string[]) => '\uFEFF' + analysisCsvRow(cells); + +// The same descriptors generate the CSV and its data dictionary. +const INTERVIEW_COLUMNS = [ + ['schema', 'schema', 'Version identifier: openinterviewer.analysis.v1.'], + ['study_id', 'study.id', 'Stored study ID.'], + ['study_name', 'study.name', 'Study name recorded with this interview.'], + ['study_revision', 'study.revision', 'Collection revision, not the current study revision.'], + ['research_question', 'study.researchQuestion', 'Research question in collectionConfig.'], + ['core_questions', 'study.coreQuestions', 'Collection core questions; JSON array in this CSV cell.'], + ['topic_areas', 'study.topicAreas', 'Collection topic areas; JSON array in this CSV cell.'], + ['interview_languages', 'study.interviewLanguages', 'Offered language codes in collectionConfig; JSON array in this CSV cell.'], + ['interview_id', 'interview.id', 'Stored interview ID.'], + ['status', 'interview.status', 'Stored interview status: in_progress or completed.'], + ['created_at_utc', 'interview.createdAt', 'Start time, ISO 8601 UTC including milliseconds.'], + ['completed_at_utc', 'interview.completedAt', 'Completion time, ISO 8601 UTC including milliseconds.'], + ['duration_minutes', 'interview.durationMinutes', 'Elapsed minutes: (completedAt - createdAt) / 60000, not rounded.'], + ['language', 'interview.language', 'Recorded participant language code; absent is not inferred as English.'], + ['language_tag', 'interview.languageTag', 'BCP-47 tag from the shared language registry (zh becomes zh-Hans).'], + ['voice_input', 'interview.voiceInput', 'Mode offered: off, browser or installation, not evidence that voice was used. A known snapshot with no setting means off.'], + ['provider_commitment', 'consent.providerCommitment', 'Recorded promise: fixed or may-change; absent means not recorded.'], + ['conducted_by_provider', 'consent.conductedBy.provider', 'Provider selected for the conversation, not synthesis.'], + ['conducted_by_model', 'consent.conductedBy.model', 'Model requested for the conversation, not reported execution provenance.'], + ['consent_transport', 'consent.transport', 'Transport disclosed at consent: cloudflare-gateway, or direct when absent in storage.'], + ['consent_hash', 'consent.consentHash', 'Recorded hash of the accepted consent text.'], + ['consent_accepted_at_utc', 'consent.consentAcceptedAt', 'Recorded consent acceptance time, ISO 8601 UTC.'], + ['analysis_status', 'analysis.status', 'pending, running, complete or failed; legacy status follows the existing analysisStatus helper.'], + ['analysis_provider', 'analysis.provider', 'Recorded synthesis provider (aiProvider).'], + ['analysis_model', 'analysis.model', 'Recorded synthesis model (aiModel).'], + ['analysis_transport', 'analysis.transport', 'Synthesis transport: cloudflare-gateway, or direct when absent in storage; not proof a synthesis ran.'], + ['bottom_line', 'analysis.bottomLine', 'Saved synthesis bottom line; missing without synthesis.'], + ['themes', 'analysis.themes', 'CSV: theme names joined with "; ". JSONL: original theme objects, including frequency and any evidence/evidenceRefs.'], + ['profile_context', 'profile.context', 'Saved participantProfile.rawContext, not raw transcript text.'], +] as const; + +export type ProfileColumn = { fieldId: string; label: string; createdAt: number; interviewId: string }; +export type ProfileColumns = Map; + +/** Across studies/revisions, newest collection time wins; ties use descending interview ID. */ +export function collectProfileColumns(columns: ProfileColumns, interview: StoredInterview): void { + const definitions = new Map(interview.collectionConfig?.profileSchema.map(field => [field.id, field.label])); + const fieldIds = new Set([...definitions.keys(), ...(interview.participantProfile?.fields ?? []).map(field => field.fieldId)]); + for (const fieldId of fieldIds) { + const previous = columns.get(fieldId); + const candidate = { fieldId, label: definitions.get(fieldId) ?? DEFINITION_UNAVAILABLE, createdAt: interview.createdAt, interviewId: interview.id }; + if (!previous || candidate.createdAt > previous.createdAt + || (candidate.createdAt === previous.createdAt && candidate.interviewId > previous.interviewId)) { + columns.set(fieldId, candidate); + } + } +} +export function sortedProfileColumns(columns: ProfileColumns): ProfileColumn[] { + return [...columns.values()].sort((a, b) => a.fieldId < b.fieldId ? -1 : a.fieldId > b.fieldId ? 1 : 0); +} +/** Pack small dictionary fragments without retaining an export-wide header or README. */ +export function* analysisTextChunks(parts: Iterable): Generator { + let chunk = ''; + for (const part of parts) { + if (chunk.length + part.length > 32 * 1024) { yield chunk; chunk = ''; } + chunk += part; + } + if (chunk) yield chunk; +} +export function* analysisInterviewsHeaderParts(columns: ProfileColumn[]): Generator { + yield analysisCsvHeader(INTERVIEW_COLUMNS.map(column => column[0])).slice(0, -2); + for (const column of columns) yield ',' + analysisCsvRow([`profile:${column.fieldId} (${column.label})`]).slice(0, -2); + yield '\r\n'; +} +export function analysisInterviewsHeader(columns: ProfileColumn[]): string { + return [...analysisInterviewsHeaderParts(columns)].join(''); +} +export function analysisInterviewsRow(interview: StoredInterview, columns: ProfileColumn[]): string { + const record = analysisInterview(interview); + const cells = INTERVIEW_COLUMNS.map(([, path]) => { + let value: unknown = record; + for (const key of path.split('.')) value = (value as Record | undefined)?.[key]; + if (path === 'analysis.themes') return record.analysis.themes?.map(theme => theme.theme).join('; '); + return Array.isArray(value) ? JSON.stringify(value) : value; + }); + const fields = new Map(record.profile.fields.map(field => [field.fieldId, field.value])); + return analysisCsvRow([...cells, ...columns.map(column => fields.get(column.fieldId))]); +} + +export const TURN_COLUMNS = ['study_id', 'interview_id', 'turn_index', 'message_id', 'role', 'speaker', 'timestamp_utc', 'offset_s', 'language', 'text']; +export function analysisTurnsCsv(interview: StoredInterview): string { + return analysisCsvHeader(TURN_COLUMNS) + analysisTurns(interview).map(turn => analysisCsvRow([ + interview.studyId, interview.id, turn.index, turn.messageId, turn.role, turn.speaker, + turn.timestamp, turn.offsetSeconds, interview.interviewLanguage, turn.text, + ])).join(''); +} +export const analysisTurnsName = (position: number) => `analysis/turns/${String(position + 1).padStart(3, '0')}.csv`; +export const PROFILE_COLUMNS = ['interview_id', 'study_revision', 'field_id', 'label_at_collection', 'value', 'status']; +export function analysisProfileRows(interview: StoredInterview): string { + return analysisProfile(interview).fields.map(field => analysisCsvRow([ + interview.id, interview.studyRevision, field.fieldId, field.label, field.value, field.status, + ])).join(''); +} + +export function analysisReadme(columns: ProfileColumn[]): string { + return [...analysisReadmeParts(columns)].join(''); +} +export function* analysisReadmeParts(columns: ProfileColumn[]): Generator { + yield `# Analysis files + +${AI_SHARING_WARNING} + +These views use each interview's own saved collectionConfig and consent/provider records, never today's study configuration. See consent in JSONL for what was recorded for each participant. Unknown optional values are omitted from JSONL and blank in CSV; explicit null profile values remain null in JSONL. Missing history is not reconstructed. All timestamps are ISO UTC. + +## Files and encoding + +- interviews.jsonl: one object per interview, schema ${ANALYSIS_SCHEMA}. UTF-8, no BOM, LF record endings. JSON escapes decode to the EXACT stored turn text, including original line endings. +- interviews.csv: one row per interview, followed by a column for every observed profile field ID across the export. +- turns/NNN.csv: one turn CSV per interview, numbered in the same export order as the top-level NNN records. Sharded instead of turns.csv to bound Worker memory. Each shard has its own header and BOM. +- profile_fields.csv: one row per saved profile field value, preserving the label at collection. +- README.md: this data dictionary and sharing guidance. + +All CSVs are UTF-8 with BOM, CRLF record endings and quoted cells. Embedded text newlines are normalized to CRLF. Every text cell, including headers, passes the spreadsheet formula-injection guard; numbers are written as plain numbers. CSV text cells may start with an added apostrophe (') for that guard: use the JSONL for verbatim quotes, not CSV. + +## interviews.csv columns and JSONL fields + +${INTERVIEW_COLUMNS.map(([name, path, description]) => `- ${name} → ${path}: ${description}`).join('\n')} + +Dynamic columns have the header profile: (), ordered by field ID (code-unit order). One column per field ID observed in saved profile values or collectionConfig.profileSchema, even across studies: do not assume the same ID has the same meaning across studies. The newest interview by createdAt supplies the label from its own collectionConfig; ties use descending interview ID. Missing definitions use "${DEFINITION_UNAVAILABLE}" rather than a current or older label. Values are preserved regardless of extraction status; blank conflates absent, null and empty text in CSV. Use JSONL or profile_fields.csv for status and historical labels. + +`; + for (const column of columns) yield `- ${escapeMarkdownInline(`profile:${column.fieldId} (${column.label})`)}: saved value for field ID ${escapeMarkdownInline(column.fieldId)}.\n`; + yield ` + +## JSONL nested structures + +- study, interview, consent, analysis and profile: grouped fields defined above. consent.conductedBy groups the recorded conversation provider and model. +- profile.fields: saved profile values, each with fieldId (schema ID), label (definition in this interview's collectionConfig, or "${DEFINITION_UNAVAILABLE}"), value (string or null), status (pending, extracted, vague or refused). profile.context is rawContext. +- analysis.themes[]: theme (name), frequency (saved count), optional evidence (legacy free text), optional evidenceRefs (citation claims). Each ref has quote (claimed excerpt), turnIndex (1-based full transcript coordinate), and optional interviewId. These are claims, not verified interpretations. +- turns[]: index, messageId, role, speaker, timestamp, offsetSeconds and text, defined below. System messages are excluded, not renumbered. + +## Turn CSV columns (JSONL turns field in parentheses) + +- study_id: containing study.id. +- interview_id: containing interview.id. +- turn_index (index): 1-based position in the FULL stored transcript, matching analysis citation numbering (EvidenceRef.turnIndex). System messages are skipped without reusing their index, so gaps are intentional. +- message_id (messageId): stored message.id. +- role (role): stored user or ai. +- speaker (speaker): Participant for user; Interviewer for ai. +- timestamp_utc (timestamp): message.timestamp as ISO UTC. +- offset_s (offsetSeconds): (message.timestamp - interview.createdAt) / 1000, not rounded; may be negative. +- language: containing interview.language, blank if unrecorded. +- text (text): message.content; exact in JSONL, guarded and newline-normalized in CSV. + +## profile_fields.csv columns + +- interview_id: containing interview.id. +- study_revision: containing study.revision (collection revision). +- field_id: profile.fields[].fieldId. +- label_at_collection: profile.fields[].label from this interview's snapshot, not the newest label. +- value: profile.fields[].value; null becomes blank. +- status: profile.fields[].status, including non-extracted values. +`; +} diff --git a/src/lib/export/interviewExport.ts b/src/lib/export/interviewExport.ts index f11c092..98f6632 100644 --- a/src/lib/export/interviewExport.ts +++ b/src/lib/export/interviewExport.ts @@ -4,11 +4,12 @@ // produce the same entries with the same decompressed bytes: per interview // NNN__.json and .md, then aggregates/.json (with JSZip's // implicit aggregates/ folder entry), then summary.csv with formula-safe -// cells. The parity tests (zipStream.test.ts, api.export.durable.test.ts) +// cells, plus snapshot-only analysis files (analysisFiles.ts). The parity tests (zipStream.test.ts, api.export.durable.test.ts) // compare the two archives entry by entry. import type { StoredAggregateSynthesis, StoredInterview } from '@/types'; import type { ExplorationAnswer } from '@/lib/exploration/types'; +import { analysisCsvHeader, analysisInterviewsHeaderParts, analysisInterviewsRow, analysisJsonLine, analysisProfileRows, analysisReadmeParts, analysisTextChunks, analysisTurnsCsv, analysisTurnsName, collectProfileColumns, PROFILE_COLUMNS, sortedProfileColumns, type ProfileColumns } from './analysisFiles'; import { csvCell } from '@/lib/csv'; import { analysisStatus } from '@/lib/analysisState'; import { createZipStream, type ZipStreamLimits } from './zipStream'; @@ -25,7 +26,7 @@ export function interviewTranscriptMarkdown(interview: StoredInterview): string `# Interview Transcript`, `Study: ${interview.studyName}`, `Interview ID: ${interview.id}`, - `Date: ${new Date(interview.createdAt).toLocaleDateString()}`, + `Date: ${new Date(interview.createdAt).toISOString()}`, `Duration: ${Math.round((interview.completedAt - interview.createdAt) / 1000 / 60)} minutes`, ...(interviewLanguageLine(interview) ? [interviewLanguageLine(interview)!] : []), `` @@ -53,7 +54,7 @@ export function interviewTranscriptMarkdown(interview: StoredInterview): string lines.push(``); interview.transcript.forEach(msg => { - const time = new Date(msg.timestamp).toLocaleTimeString(); + const time = new Date(msg.timestamp).toISOString(); const role = msg.role === 'user' ? 'PARTICIPANT' : 'INTERVIEWER'; lines.push(`[${time}] ${role}:`); // Quoted so transcript text cannot imitate a speaker line or a heading. @@ -142,7 +143,8 @@ export class ExportSnapshotChangedError extends Error { } export type InterviewExportStreamInput = { - pages: AsyncIterable; + /** A fresh walk of the SAME snapshot on every invocation; never begin a new export. */ + pages: () => AsyncIterable; /** * Called after the last entry and before the central directory is written. * Throw (e.g. ExportSnapshotChangedError) to fail the archive instead. @@ -159,23 +161,28 @@ export type InterviewExportStreamInput = { * sequence check, ZIP limit) errors the stream; the central directory is * only written after every page and `beforeFinish` succeeded, so a client * can never mistake a truncated download for a complete archive. - * summary.csv is buffered as one short line per interview (the export's - * interview ceiling bounds it); every other entry is written as it arrives. + * Replay the same bounded pages for combined analysis entries: no corpus-sized + * strings or arrays are retained. Only the profile column dictionary survives a pass. */ export function createInterviewExportStream(input: InterviewExportStreamInput): ReadableStream { const zip = createZipStream({ modifiedAt: input.modifiedAt, limits: input.limits }); const run = async () => { - const csvLines = [SUMMARY_CSV_HEADER]; + const columns: ProfileColumns = new Map(); + await zip.addDirectory('analysis/'); + await zip.addDirectory('analysis/turns/'); const seenAggregates = new Set(); const seenExplorations = new Set(); let position = 0; - for await (const page of input.pages) { + const interviewIds: string[] = []; + for await (const page of input.pages()) { for (const interview of page.interviews) { const baseName = interviewEntryBaseName(position, interview); position += 1; + interviewIds.push(interview.id); await zip.addFile(`${baseName}.json`, interviewJson(interview)); await zip.addFile(`${baseName}.md`, interviewTranscriptMarkdown(interview)); - csvLines.push(summaryCsvRow(interview)); + await zip.addFile(analysisTurnsName(position - 1), analysisTurnsCsv(interview)); + collectProfileColumns(columns, interview); } for (const aggregate of page.aggregates) { if (seenAggregates.size === 0) await zip.addDirectory(AGGREGATES_DIRECTORY); @@ -191,7 +198,27 @@ export function createInterviewExportStream(input: InterviewExportStreamInput): await zip.addFile(entryName, explorationJson(answer)); } } - await zip.addFile(SUMMARY_CSV_NAME, csvLines.join('\n')); + const orderedColumns = sortedProfileColumns(columns); + // Count and identity checks catch a broken replay source as well as missing pages. + // Durable storage itself checks the snapshot fingerprint on each page read. + async function* rows(header: string | Iterable, row: (interview: StoredInterview) => string): AsyncGenerator { + if (typeof header === 'string') yield header; + else yield* header; + let count = 0; + for await (const page of input.pages()) { + for (const interview of page.interviews) { + if (interview.id !== interviewIds[count]) throw new ExportSnapshotChangedError(); + count += 1; + yield row(interview); + } + } + if (count !== position) throw new ExportSnapshotChangedError(); + } + await zip.addFile(SUMMARY_CSV_NAME, rows(SUMMARY_CSV_HEADER, interview => `\n${summaryCsvRow(interview)}`)); + await zip.addFile('analysis/interviews.jsonl', rows('', analysisJsonLine)); + await zip.addFile('analysis/interviews.csv', rows(analysisTextChunks(analysisInterviewsHeaderParts(orderedColumns)), interview => analysisInterviewsRow(interview, orderedColumns))); + await zip.addFile('analysis/profile_fields.csv', rows(analysisCsvHeader(PROFILE_COLUMNS), analysisProfileRows)); + await zip.addFile('analysis/README.md', analysisTextChunks(analysisReadmeParts(orderedColumns))); if (input.beforeFinish) await input.beforeFinish(); await zip.finish(); }; diff --git a/src/lib/export/transcriptsMarkdown.ts b/src/lib/export/transcriptsMarkdown.ts index 37609e1..501212f 100644 --- a/src/lib/export/transcriptsMarkdown.ts +++ b/src/lib/export/transcriptsMarkdown.ts @@ -12,6 +12,13 @@ import type { StoredInterview, StoredStudy, StudyConfig } from '@/types'; import { PROVIDER_MODELS, PROVIDER_OPTIONS } from '@/lib/providerRegistry'; import { LANGUAGE_ENGLISH_NAMES, LANGUAGE_NATIVE_NAMES } from '@/lib/i18n/languages'; +export const AI_SHARING_WARNING = [ + '> **Before sharing this file with an AI tool:** each interview below records what its', + '> participant was told about the AI that would handle their responses. Where a participant', + '> was told the study uses only one provider and model, sending this file to a different AI', + '> service may break that promise.', +].join('\n'); + export const TRANSCRIPTS_MARKDOWN_CONTENT_TYPE = 'text/markdown; charset=utf-8'; const COMPLETE_MARKER_PREFIX = '