Skip to content

fix(libs): reject non-dict output in parse_and_check_json_markdown - #45225

Open
TINGyu123644 wants to merge 1 commit into
langgenius:mainfrom
TINGyu123644:fix/json-parser-scalar-validation
Open

TINGyu123644 wants to merge 1 commit into
langgenius:mainfrom
TINGyu123644:fix/json-parser-scalar-validation

Conversation

@TINGyu123644

Copy link
Copy Markdown

Fixes #45224

Problem found

parse_and_check_json_markdown in api/libs/json_in_md_parser.py only special-cases list results before running the key not in json_obj check, so non-dict objects leak into it with two wrong behaviours:

  1. A valid fenced JSON scalar (```json\n42\n```, null, true — the fence fallback in parse_json_markdown deliberately supports non-bracket content, and an existing unit test pins fenced scalars as supported) raises TypeError: argument of type 'int' is not iterable instead of the contract OutputParserError.
  2. A valid JSON string passes validation whenever an expected key appears inside it as a substring (str.__contains__ semantics), and is returned to callers as if it were the parsed object. RuleConfigGeneratorOutputParser.parse then crashes on parsed["prompt"] with TypeError: string indices must be integers.

The parser consumes raw LLM output, so these inputs are realistic. Existing callers catch broad Exception, so today this degrades the feature rather than crashing the process, but the error contract is broken and case 2 skips validation entirely.

Reproduction

Run against main (before this patch):

from libs.json_in_md_parser import parse_and_check_json_markdown

parse_and_check_json_markdown("```json\n42\n```", ["answer"])
# TypeError: argument of type 'int' is not iterable  (expected OutputParserError)

parse_and_check_json_markdown('```json\n"the generated prompt text"\n```', ["prompt"])
# returns "the generated prompt text"  (expected OutputParserError)

Fix

After the single-element list unwrapping, enforce isinstance(json_obj, dict) and raise OutputParserError(f"got invalid return object. obj:{json_obj}") otherwise — same error shape the list branch already uses. Behaviour for dict (and single-dict list) results, and for unfenced/unparseable text, is unchanged.

Verification

Local sandbox with the package structure mirrored (libs.json_in_md_parser + core.llm_generator.output_parser.errors stub, pytest via uv):

  • 4 new regression tests (fenced 42 / null / true → OutputParserError; JSON string containing an expected key substring → OutputParserError): failed on the unpatched code (3 × TypeError, 1 × no error raised), all pass after the patch.
  • The existing api/tests/unit_tests/libs/test_json_in_md_parser.py suite (13 tests): all pass unchanged after the patch (no regression).

Diff: 1 file changed in the parser (4 lines added), regression tests appended to the existing test module.

The change was drafted with an AI coding assistant and reviewed before submission.

parse_and_check_json_markdown only special-cases lists; any other
non-dict object falls through to the `key not in json_obj` check:

- a valid JSON scalar (e.g. a fenced `42`, `null` or `true`, which
  parse_json_markdown explicitly supports via its fence fallback)
  raises TypeError instead of the contract OutputParserError;
- a valid JSON string passes validation via substring matching
  whenever an expected key appears inside the string, and is returned
  to callers that then crash indexing it.

Enforce that the parsed object is a dict (or a single-element list
wrapping a dict) and raise OutputParserError otherwise. Verified
against the existing unit tests plus new regression tests.
@github-actions github-actions Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parse_and_check_json_markdown raises TypeError (or silently passes validation) on non-dict LLM output

1 participant