Skip to content

test(corpus): calibrate notification-t case expectations against live… - #202

Merged
zhangwenjun-huawei merged 1 commit into
project-openan:devfrom
z30048074:dev
Oct 19, 2026
Merged

zhangwenjun-huawei merged 1 commit into
project-openan:devfrom
z30048074:dev

Conversation

@z30048074

@z30048074 z30048074 commented Oct 17, 2026 •

Copy link
Copy Markdown

… LLM

Tune the Notification-T corpus cases after a real-LLM run so the live suites pass deterministically:

  • assert only the deterministic topic field on success; drop the exact-match assertions on subscriptionCondition/notificationDataFormat whose punctuation the LLM normalizes non-deterministically (为/为: and trailing 。 vary per run)
  • re-categorize the incomplete-format cases (service-recovery TC00000003) from 正常 to 异常, expecting negotiation.semantic_rejected, since the semantic validator correctly rejects the missing required items
  • switch the subscribe-incident fault-priority wording from English to Chinese (critical、high triggers content.value_not_allowed in the text flow while the data flow accepts it)
  • drop the from-text slot.not_provided case (the LLM fabricates the required 上报通知数据格式 slot value non-deterministically); keep the deterministic from-data variant

Description

Related Issue(s)

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Other (describe below)

Checklist

  • I have read the CONTRIBUTING guide
  • All commits are signed off (git commit -s) per the DCO
  • New and existing tests pass locally
  • I have added tests that prove my fix is effective or my feature works (if applicable)
  • I have added or updated documentation as needed
  • New source files include the SPDX license header

Additional Context

… LLM

Tune the Notification-T corpus cases after a real-LLM run so the live suites
pass deterministically:

- assert only the deterministic topic field on success; drop the exact-match
  assertions on subscriptionCondition/notificationDataFormat whose punctuation
  the LLM normalizes non-deterministically (为/为: and trailing 。 vary per run)
- re-categorize the incomplete-format cases (service-recovery TC00000003) from
  正常 to 异常, expecting negotiation.semantic_rejected, since the semantic
  validator correctly rejects the missing required items
- switch the subscribe-incident fault-priority wording from English to Chinese
  (critical、high triggers content.value_not_allowed in the text flow while the
  data flow accepts it)
- drop the from-text slot.not_provided case (the LLM fabricates the required
  上报通知数据格式 slot value non-deterministically); keep the deterministic
  from-data variant

Signed-off-by: z30048074 <zouwenchong@h-partners.com>
@z30048074
z30048074 requested a review from a team as a code owner October 17, 2026 10:31
@zhangwenjun-huawei
zhangwenjun-huawei merged commit 63be3ec into project-openan:dev Oct 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants