Skip to content

fix(import): fail the metadata import on over-length free text - #992

Open
jacobvjk wants to merge 1 commit into
epic/v2from
fix/metadata-import-text-limits
Open

jacobvjk wants to merge 1 commit into
epic/v2from
fix/metadata-import-text-limits

Conversation

@jacobvjk

Copy link
Copy Markdown
Collaborator

Closes #990.

The metadata importer used to drop most free text that was over its character limit, and the only sign was a line in the report. Now every limit is an authoring error, so nothing is written until the workbook is fixed. The limits are the ones the cookbook now states in RMI/tpr-tpc#37 (decision 0028).

Tab › column Limit Before Now
draft_core_drivers › all 7 drivers 500 note, value left null error
draft_dependencies › all 10 descriptions 500 note, dependency skipped error
draft_data_availability › Scope limitations 500 error error (counted the same way as the others)
draft_metadata › Pathway Overview 2,500 note, existing value kept error
draft_metadata › Description 100 note, template description used error
draft_metadata › Name of Pathway 200 whole cell, 20 inside the brackets not checked error
draft_metadata › Name of publication 200 not checked error

Worth knowing

  • New pathways only: Description, Name of Pathway and Name of publication are checked only for new pathways. An update keeps the name, description and publication already in its JSON and never reads those cells. The cookbook validator (R2.5) still checks them upstream.
  • How characters are counted: in code points, the way the cookbook and JSON-schema maxLength count them. CO₂ is 3 characters and an emoji is 1. JavaScript's .length would count the emoji as 2.
  • What is measured: the text as written, after the importer adds a final period. A 500-character cell with no period becomes 501 and fails, which schema:check would reject anyway. This is one character stricter than the cookbook validator, which measures the raw cell.
  • v1 fallback unchanged: a pathwayDescription taken from an old v1 expertOverview still only gets a note when it is over 2,500. That text isn't in the workbook, so there's no cell to point the author to.
  • First import after merging may fail: the next import can stop on cells that used to be dropped silently. Each error names the cell to shorten.

Verification

  • npx vitest run scripts/import-pathway-data.test.ts: 35 passing. 9 are new and cover each limit at exactly the limit and one over, the added period, code-point counting, and an update with over-length cells that must not error.
  • eslint and prettier are clean on both files. tsc -p tsconfig.scripts.json reports 5 errors in these files, all on unchanged lines and all present on epic/v2 before this change.
  • @jacobvjk ran a dry run against the current pathway_data_prepared.xlsx and it behaved as expected.

🤖 Generated with Claude Code

Most free-text cells over their limit were dropped silently: a core driver
left null, a dependency skipped, a new pathway's description replaced by the
template's, an over-length Pathway Overview ignored. Every limit is now an
authoring error, so nothing is written until the workbook is fixed.

Limits follow the cookbook (RMI/tpr-tpc#37, decision 0028):
- Description 100, Name of publication 200 (new pathways only)
- Name of Pathway 200 whole cell, 20 inside the brackets (new pathways only)
- Pathway Overview 2,500
- each core driver and dependency description 500
- Scope limitations 500 (already an error; now counted the same way)

Length is counted in code points, as the cookbook and JSON-schema maxLength
do, on the value as written (after the trailing period is added).

Closes #990

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Expected version change and release notes

🚨 WARNING: This PR is not expected to trigger a new version

To trigger a version bump, use at least one conventional commit message in this branch. See: https://www.conventionalcommits.org/en/v1.0.0/

@github-actions

Copy link
Copy Markdown

Azure Static Web Apps: Your stage site is ready! Visit it here: https://proud-glacier-0f640931e-992.westus2.2.azurestaticapps.net

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Target filtering can skip over-length pathway abbreviations before validation, allowing the import to succeed.

1 open finding
What changed in this PR

Makes over-length workbook text block metadata imports instead of being silently dropped, aligning limits with cookbook decision 0028.

Changes:

  • Adds shared code-point counting and length validation.
  • Applies limits across metadata fields while preserving documented exceptions.
  • Adds boundary, punctuation, Unicode, and update-exemption tests.
File Description
scripts/​import-pathway-data.ts Reports over-length imported text as blocking errors.
scripts/​import-pathway-data.test.ts Tests character limits and validation exceptions.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/import-pathway-data.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants