fix: make test suite and tools portable across default encodings and python -OO (#484)

[drk-ram]

What this changes

Closes #484.

Contributor: Raghuvar Durga (@drk-ram)

Makes the test suite, conformance suite, fixture generators, and repository tooling portable across non-UTF-8 default encodings (LC_ALL=C PYTHONUTF8=0 / Windows cp1252) and python -OO (PYTHONOPTIMIZE=2), and raises the referencing dependency floor so it is satisfiable alongside jsonschema>=4.26.0, per @imran-siddique's scope ruling on #484:

1. Explicit UTF-8 file and subprocess encodings:
   • Adds encoding="utf-8" to all bare Path.read_text() and Path.write_text() calls across tests/ (26 files), conformance/tests/ (8 files, including conformance/tests/test_manifest_ref_draft_vectors.py and conformance/tests/vectors/manifest-ref-v0.3-draft/generate.py with newline="\n"), and examples/verifier-token-conformance/gen_corpus.py.
   • Adds encoding="utf-8" to subprocess.run(..., text=True) in tools/review_bar.py so git diff -z path output and pytest output decode as UTF-8 under LC_ALL=C PYTHONUTF8=0.
   • Cleans four em dashes in docstrings of conformance/tests/test_measurement_harness.py so python tools/check_dashes.py continues to pass once that file is edited past its conformance/import-manifest.json snapshot.
2. Guard __doc__ under python -OO (PYTHONOPTIMIZE=2):
   • Guards description=(__doc__ or "").split("\n", 1)[0] in tools/review_bar.py and description=(__doc__ or "").splitlines()[0] if __doc__ else "" in tools/check_dashes.py.
3. Raise referencing floor to >=0.28.4:
   • Updates referencing>=0.28 to referencing>=0.28.4 in pyproject.toml and requirements/dev.in to match jsonschema>=4.26.0's requirement.
4. CI portability check:
   • Adds a step in .github/workflows/ci.yml running LC_ALL=C PYTHONUTF8=0 pytest and PYTHONOPTIMIZE=2 pytest tests/test_review_bar.py.

Cold-reader review:

• Ran an independent code-review sub-agent over the full diff against the Test suite and tools are not portable across default encodings and python -OO; referencing floor is unsatisfiable #484 scope and verified via AST inspection that zero bare .read_text() or .write_text() calls remain anywhere in the repository; the cold reader reported no issues.
• Running tests/test_review_bar.py under PYTHONUTF8=0 during verification exposed that tools/review_bar.py's git() helper also used subprocess.run(..., text=True) without encoding="utf-8", causing test_a_test_path_git_would_quote_is_still_found to misdecode UTF-8 paths emitted by git diff -z; added encoding="utf-8" there as well.

Type of change

• Editorial (typo, link fix, clarification: no normative effect)
• Non-breaking spec change (new optional field, new platform profile, informative addition)
• Breaking spec change (requires 14-day comment period and Project Lead sign-off)
• Schema change
• Example addition

Spec section

None (test suite, tooling, CI workflow, and package dependency metadata only).

Tests for a fix

Not a library or schema verification fix (src/agentrust_trace/ and schema/ are unchanged), but the portability fixes are directly exercised by existing tests that fail on main when the changes are switched off:

• PYTHONUTF8=0 pytest tests/test_regex_surface_parity.py tests/test_schema_regex_semantics.py tests/test_build_provenance_depth_doc.py and PYTHONUTF8=0 pytest conformance/tests/test_manifest_ref_draft_vectors.py fail with UnicodeDecodeError / fixture mismatch without the explicit encoding="utf-8" reads and pass with them (conformance/tests: 778 passed, 5 xpassed; root tests/: 3,734 passed).
• PYTHONOPTIMIZE=2 pytest tests/test_review_bar.py -k "test_the_command_line_writes_the_summary_and_never_fails_the_build or test_an_error_inside_the_tool_still_writes_a_report_and_exits_zero" and python -OO tools/check_dashes.py fail with AttributeError: 'NoneType' object has no attribute 'split' / 'splitlines' without the __doc__ guards and pass with them.
• ruff check src tests scripts, python tools/check_dashes.py, and mypy src/agentrust_trace all pass.

Checklist

• DCO sign-off on all commits (git commit -s)
• CHANGELOG.md updated (for any normative change)
• Breaking changes marked with <!-- CHANGED: #NNN: description --> in spec text
• Backward compatibility statement included (for breaking changes)

[rajnisht7]

Encoding fix looks good, fixing encoding in several conformance/tests/ files (conftest, anchor/manifest-ref vectors, measurement harness, unit loader/runner/CLI, generator). The new CI step is:

LC_ALL=C PYTHONUTF8=0 pytest
PYTHONOPTIMIZE=2 pytest tests/test_review_bar.py

Root pyproject.toml has testpaths = ["tests"], so that only runs root tests/. conformance-ci.yml still runs under a normal UTF-8 locale. So the conformance portability fixes aren’t regression-protected.
kindly add a non-UTF-8 run for conformance as well (in ci.yml or conformance-ci.yml), e.g. something that actually executes conformance/tests under LC_ALL=C PYTHONUTF8=0.

Also minor:

• PYTHONOPTIMIZE=2 only exercises test_review_bar.py. tools/check_dashes.py was also fixed for -OO but isn’t run under -OO in CI, worth adding PYTHONOPTIMIZE=2 python tools/check_dashes.py.
• In review_bar.py, prefer description=(__doc__ or "").split("\n\n")[0] so the -OO guard doesn’t change the existing “first paragraph” split behavior.

[imran-siddique]

Approving 68b778c. This is the scope agreed on #484. The red test (3.11) and (3.12) on 8803282 was the new LC_ALL=C job doing its job on the non-ASCII filename test, and 68b778c skips that one case. On Windows cp1252 the git-quoted path test fails on #537, #538 and #539 heads and passes here. python -OO tools/check_dashes.py --help and tools/review_bar.py --help crash with AttributeError on 2bbecec and work here. Ruff and check_dashes are clean.

[imran-siddique]

The portability job is now the one place this test never runs. Fine, but say so in the skip reason.

[imran-siddique]

(__doc__ or "") is redundant next to if __doc__ else "".

[imran-siddique]

This also changes the help text from the first paragraph to the first line. Intended?