[witbrock]

What changed

Implements JVNAUTOSCI-2592 as an end-to-end, model-led spreadsheet-to-programme representation capability:

• binds an authenticated uploaded file copy to the exact chat request as structured workflow input
• extracts bounded XLSX evidence with row/cell/formula provenance
• compiles model-authored relational plans into stable records and dataset-scoped identities
• enforces additive, idempotent KR materialisation through deterministic write-authority guards
• reconciles added, changed, unchanged, ambiguous and missing source rows without destructive deletion
• publishes represented batch/item workflows and planning authority from versioned seed artefacts
• strengthens durable workflow propagation, read-back and failure handling
• improves Atlas deadlines, batching, extent-index degradation/fallback and observability
• records Atlas efficiency as a standing repository engineering priority

Why

The PhD supervision workbook needs to become trustworthy, provenance-bearing programme knowledge without hard-coding its domain mapping into Python, leaking private data, merging ambiguous people, or duplicating state on re-upload/retry.

Impact

Users can upload a workbook in Von and request representation through the normal chat path. Semantic mapping remains model/represented-workflow driven while code owns authentication, parsing limits, idempotency, provenance, safety guards and canonical read-back.

The private reference workbook and student data are not included in this PR.

Validation

• 292-test integrated backend run passed during implementation
• focused merge gate: 59 backend tests passed
• frontend upload-binding gate: 5 tests passed
• additional workflow/runtime, Atlas/privacy and frontend suites passed during implementation
• Ruff/static JS checks, seed exactness, workflow-purity report and git diff --check passed
• staged privacy/secret scan found no private workbook data or credentials

Live private-data import and the five-reset-state certification campaign remain post-merge acceptance work through the authenticated Von UI.

[Copilot]

Pull request overview

Implements an end-to-end, model-led spreadsheet → represented-programme capability by (a) binding an authenticated uploaded file-copy concept to a chat request, (b) extracting bounded XLSX evidence, (c) compiling a model-authored record plan into stable dataset/record identities, and (d) enforcing additive/idempotent KR materialisation with deterministic write-guarding and read-back.

Changes:

• Adds represented spreadsheet programme workflow + prompt seed support (bootstrap, discovery, routing, and required-effects/read-back contracts).
• Introduces bounded structured spreadsheet evidence extraction and deterministic spreadsheet-record internal MCP tools to compile/compare/receipt batches and records.
• Strengthens safety/operational behaviour: optional KR materialisation guard action, durable checkpoint projection guarantees for declared reads, stricter attachment/namespace binding, and LLM raw-IO suppression + request timeout handling.

Reviewed changes

Copilot reviewed 75 out of 76 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/backend/test_workflow_transition_conditions.py	Verifies extent-index sync is deferred until workflow run completion.
tests/backend/test_workflow_purity_report.py	Updates purity allowlist for new workflow bootstrap service.
tests/backend/test_workflow_metadata_validation.py	Adds coverage for dotted context-path reads in metadata validation.
tests/backend/test_workflow_execution_contracts.py	Ensures explicit “failed” terminal states are treated as failure.
tests/backend/test_workflow_definition_identity_service.py	Allows for_each.items to be mapped from context via $context_key.
tests/backend/test_workflow_concept_authority_service.py	Adjusts mock signature to tolerate extra kwargs.
tests/backend/test_vontology_loader.py	Adds tests for discovery short-circuiting and lifecycle text prefetch fallback.
tests/backend/test_von_generate_attachment_binding.py	Tests generate-route attachment binding/authorisation for file copies.
tests/backend/test_turn_execution_selected_workflow_output_propagation.py	Ensures required-effects tools for unchanged spreadsheet replay.
tests/backend/test_spreadsheet_programme_workflow_vontology_service.py	End-to-end tests for spreadsheet workflow bootstrap + prompt + idempotency.
tests/backend/test_spreadsheet_model_selected_table_region.py	Tests table-region selection + exclusion reasoning requirements.
tests/backend/test_spreadsheet_cross_record_entity_identity.py	Tests dataset-scoped entity identity reuse and guard enforcement.
tests/backend/test_source_processing_marker_service.py	Adds tests for marker existence lookup, fingerprint/authority reconciliation, and represented-output checks.
tests/backend/test_read_file_copy_tool.py	Adds namespace-mismatch rejection + structured spreadsheet evidence extraction tests.
tests/backend/test_orchestrator_structured_calling.py	Verifies timeout threading into structured calling request params.
tests/backend/test_openai_structured_tool_transport.py	Adds fake client with_options support + timeout-as-transport-option test.
tests/backend/test_mongo_collection_index_guards.py	Expands index coverage assertions for text-relations lookups.
tests/backend/test_local_model_preflight.py	Ensures sensitive calls bypass raw-response cache when IO suppression is active.
tests/backend/test_llm_step_executor.py	Tests raw-IO suppression propagates across timeout worker thread.
tests/backend/test_kr_materialisation_workflow_vontology_service.py	Verifies seed uses prompt concepts, applies guard steps, and suppresses raw IO.
tests/backend/test_kr_materialisation_guard_service.py	Adds direct tests for optional guard acceptance/rejection behaviour.
tests/backend/test_internal_mcp_catalogue_builds.py	Verifies spreadsheet record tools are registered and bounded as internal surfaces.
tests/backend/test_file_copy_upload_classification_workflow.py	Adds explicit spreadsheet route selection tests and avoids mutation by format alone.
tests/backend/test_file_copy_typing_service.py	Adds spreadsheet-by-format routing score/type coverage.
tests/backend/test_durable_workflow_executor_safety.py	Confirms durable failure terminal semantics and preserves declared execution inputs in checkpoints.
tests/backend/test_control_flow_actions.py	Tests KR guard action registration + empty for_each invocations behaviour.
tests/backend/test_concepts_repository_query_deadline.py	Ensures repository find applies server deadline before cursor options.
tests/backend/test_concept_search_service_performance.py	Enforces query deadlines + CSOT consumption boundaries for concept search.
tests/backend/test_auto_names.py	Ensures concept creation does not log private display names.
tests/backend/fixtures/workflow_purity_baseline.json	Updates baseline counts/timestamp after new capability additions.
src/workflows/von/main.py	Suppresses additional noisy SDK loggers to avoid leaking sensitive request bodies.
src/frontend/web/von_interface/static/js/chatTab.js	Binds uploaded file-copy IDs to the correct chat request/session and preserves on retry/abort.
src/backend/workflows/workflow_purity_report.py	Allows the new spreadsheet workflow bootstrap service in purity reporting.
src/backend/workflows/workflow_definition_identity_service.py	Accepts for_each.items via $context_key mapping in contract validation.
src/backend/workflows/workflow_concept_authority_service.py	Adds reads_context_keys to publication spec and round-trips it into definitions.
src/backend/workflows/vontology_loader.py	Avoids unsafe empty-cache on lifecycle prefetch failure; adds lifecycle text prefetch for candidates.
src/backend/workflows/repo_seed_bundles/workflow_template_seed_bundle.json	Switches template selection mode to explicit-only.
src/backend/workflows/repo_seed_bundles/spreadsheet_programme_plan_prompt_seed.md	Adds the spreadsheet planning prompt seed (untrusted evidence, idempotency, and vocabulary constraints).
src/backend/workflows/repo_seed_bundles/prompt_kr_design_materialisation_plan_seed.md	Adds explicit instructions for respecting optional materialisation guards.
src/backend/workflows/repo_seed_bundles/kr_materialisation_workflow_seed_bundle.json	Adds optional guard action, improves exact read-back evidence tooling, and updates seed version.
src/backend/workflows/repo_seed_bundles/canonical_workflow_publication_seed_bundle.json	Adds spreadsheet route to typed subworkflow route-map and bumps seed version.
src/backend/workflows/metadata_validation.py	Allows dotted-path reads in metadata validation.
src/backend/workflows/llm_step_executor.py	Propagates contextvars into timeout threads and enforces raw-IO suppression via represented policy.
src/backend/workflows/execution_contracts.py	Adds KR guard action id and treats failure-like terminal states as failed.
src/backend/workflows/engine.py	Defers extent-index sync for the whole run and treats explicit failed terminals as failure.
src/backend/workflows/durable/file_copy_upload_classification_workflow.py	Adds spreadsheet as an allowed forced route key.
src/backend/workflows/durable/durable_executor.py	Defers extent-index sync, preserves execution-required checkpoint keys, and honours failure-like terminals.
src/backend/workflows/durable/control_flow_actions.py	Adds KR materialisation guard action and ensures for_each emits invocations consistently.
src/backend/workflows/durable/checkpoint_context_projection.py	Adds lossless_keys to preserve declared execution-required context through checkpointing.
src/backend/services/workflow_vontology_materialisation_helpers.py	Defers extent-index sync during workflow seed materialisation with event integration suspended.
src/backend/services/workflow_repo_seed_bootstrap.py	Suspends event workflow integration while materialising support concepts.
src/backend/services/text_value_service.py	Uses batch access filtering for concept text retrieval.
src/backend/services/spreadsheet_programme_workflow_vontology_service.py	Bootstraps spreadsheet programme workflow family + prompt + dependencies with fail-closed parent verification.
src/backend/services/source_processing_marker_service.py	Adds bounded existence lookup, fingerprint/authority tracking, and spreadsheet output requirements.
src/backend/services/file_copy_typing_service.py	Adds spreadsheet format typing and keeps spreadsheet score below auto-mutation threshold.
src/backend/services/concept_service.py	Removes logging of sensitive display names during concept creation.
src/backend/services/concept_search_service.py	Adds server deadlines and CSOT for repository reads; centralises cursor consumption.
src/backend/server/utils_flask.py	Adds spreadsheet workflow bootstrap to durable workflow system bootstrap sequence.
src/backend/server/routes/von_routes.py	Adds authenticated file-copy input authorisation for generate workflow inputs.
src/backend/languagemodels/structured_tool_calling/providers/openai_client.py	Treats request timeout as transport option via with_options, not a model parameter.
src/backend/languagemodels/llm_interface.py	Adds contextvar-based raw-IO suppression affecting logging and response cache.
src/backend/integrations/internal_mcp/spreadsheet_record_tools.py	Adds deterministic internal MCP adapters for spreadsheet record plan/compare/receipts.
src/backend/integrations/internal_mcp/orchestrator.py	Threads timeout override into OpenAI structured-calling transport options.
src/backend/integrations/internal_mcp/catalogue.py	Adds structured spreadsheet reading, strengthens namespace enforcement, and registers new spreadsheet tools.
src/backend/db/repositories/concepts_repository.py	Adds max_time_ms support to repository find.
src/backend/db/mongo_client.py	Adds a composite index to support common text-relations lookup patterns.
AGENTS.md	Updates review date and records Atlas efficiency as a standing priority.