The first real pair said negative: rank on meaning, and let the same subject disagree - #212
Merged
Merged
Conversation
…subject disagree Two real people were driven through claude.ai and ChatGPT as ordinary clients. One had reasoned about a public registry of how employers treat candidates; the other, independently and in another domain, about a registry of how landlords treat tenants. Step for step it was the same construction: conduct is private and scattered, so the decision is blind; it costs the stronger party nothing, so it repeats; a registry makes conduct visible before the decision and deters it; the real constraint is the cold start, which needs an organisation that already holds the histories. The engine returned negative, and returned a template coincidence above it. Measured on production: the coincidence at structural 0.305 / semantic 0.120, the twin at 0.186 / 0.590 / contradiction 0.214. Four defects, none of them visible to the suite or to Benchmark v0.2. Ranking read one component. `_verified_sort_key` ordered by `structural` alone, so shape without meaning led the list -- and shape alone is exactly what cannot tell a coincidence from a match, which is why `semantic` is computed at all. It now ranks on 0.65 structural + 0.35 semantic, with structural as the tie-break. Classification is untouched: a coincidence is still negative, it just no longer leads. The same-subject branch demanded contradiction == 0.0 exactly. The pair had 0.214, from two crossed correspondences -- one person's "learning after the fact" laid over the other's "no cost to the landlord", and the reverse. That is not noise. It is a disagreement about which is the first cause, and when the two were introduced and asked, the second answered that the root is the asymmetry of power at the moment of the deal, not the invisibility of the information. The engine had used the most interesting thing about the pair as its reason to hide them from each other. Same subject now buys a higher contradiction ceiling, not immunity; polarity conflicts are still hard-rejected by h_sign_conflict. Policy .../same-subject/0.4, ADR-0007. The tool contract demanded English labels, so an assistant driving a Russian conversation translated the reasoning first -- and said so, honestly downgrading authorship to their_words_reorganised, because a translation is the assistant's phrasing and it is that phrasing which then gets matched. It also defeated the multilingual encoder the deployment actually runs. The contract now says the index compares meaning across languages and names any script it cannot read. Written declaratively: the descriptions say what the server does rather than instructing the model, which is a rule the suite enforces and my first draft broke. The chat drawing was not the page's drawing. It placed dots on rings by structural score, so it told a different story from the list beside it, and it printed one pseudonym twice when a person had two matching thoughts with no way to tell which dot was which. It is now the same seven-axis radar the page draws -- same axes, same order, same two inversions -- one polygon per match, each legend row naming the thought. Benchmark v0.2 passes unchanged on both splits with gold unedited, including same_vocabulary_wrong_structure, the family designed to catch exactly this loosening. The gap was that the benchmark has no case for two people on the same subject who order the causes differently. Production produced one, and it is now tests/test_same_subject_resonance.py, built from the measured components rather than from authored fixtures. Not fixed, and now stated plainly in the contract and the roadmap: prose extraction is English-only, down to its sentence splitting and clause boundaries. A Russian conversation yields an honest empty graph and the assistant supplies the structure instead. Making it multilingual is a real mission with its own gate, not more regular expressions. 690 tests OK, both gates pass with gold unedited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two real people, driven through claude.ai and ChatGPT as ordinary clients. One had reasoned about a registry of how employers treat candidates; the other, independently and in another domain, about a registry of how landlords treat tenants. Step for step it was the same construction — including the cold start and the need for an organisation that already holds the histories.
The engine returned
negative, and put a template coincidence above it.Four defects, none visible to the suite or to Benchmark v0.2.
1. Ranking read one component.
_verified_sort_keyordered bystructuralalone — and shape alone is exactly what cannot tell a coincidence from a match, which is whysemanticis computed. Now0.65·structural + 0.35·semantic. Classification untouched.2. The same-subject branch demanded
contradiction == 0.0exactly. The 0.214 came from two crossed correspondences. That is a disagreement about which is the first cause — and when the two people were introduced and asked, the second answered that the root is the asymmetry of power at the moment of the deal, not the invisibility of the information. The engine had used the most interesting thing about the pair as its reason to hide them from each other. Same subject now buys a higher ceiling (0.35), not immunity;h_sign_conflictstill hard-rejects real polarity conflicts.3. The contract demanded English labels. An assistant driving a Russian conversation translated first and honestly downgraded authorship to
their_words_reorganised— a translation is the assistant's phrasing, and it is that phrasing which then gets matched. It also defeated the multilingual encoder production runs.4. The chat drawing was not the page's drawing. Dots on rings by structural score, telling a different story from the list beside it, and printing one pseudonym twice for two thoughts. It is now the same seven-axis radar the page draws — same axes, same order, same two inversions — one polygon per match, each row naming the thought.
Why this is not tuning the gate away
Benchmark v0.2 passes unchanged on both splits, gold unedited: accuracy 1.0, negative FPR 0.0, polarity rejection 1.0, Recall@5/@20 1.0.
same_vocabulary_wrong_structure— the family designed to catch exactly this loosening — stays negative.The gap was that the benchmark has no case for two people on the same subject who order the causes differently. Production produced one. It is now
tests/test_same_subject_resonance.py, built from the measured components, not authored fixtures. Recorded in ADR-0007.Not fixed, stated plainly
Prose extraction is English-only, down to its sentence splitting and clause boundaries. A Russian conversation yields an honest empty graph and the assistant supplies the structure instead. Multilingual extraction is a real mission with its own gate — now §5 of the roadmap.
🤖 Generated with Claude Code