Skip to content

Fix character-limit chunking for overlong tokens - #7

Open
blue126 wants to merge 1 commit into
sharadcodes:mainfrom
blue126:fix/overlong-token-chunking
Open

Fix character-limit chunking for overlong tokens#7
blue126 wants to merge 1 commit into
sharadcodes:mainfrom
blue126:fix/overlong-token-chunking

Conversation

@blue126

@blue126 blue126 commented Sep 8, 2026

Copy link
Copy Markdown

Fixes #6.

Summary

  • hard-split a token that exceeds the configured character limit instead of rejecting the whole audiobook
  • use the same rule in the frontend chunk-count estimator
  • add backend and frontend regression coverage

Validation

  • uv run ruff check src tests
  • uv run mypy src
  • uv run pytest -q (48 passed)
  • npm test -- --runInBand (6 passed)
  • npm run build

This keeps every Character limit chunk at or below the provider cap, including URLs, OCR runs, and CJK text without whitespace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Character-limit chunking rejects EPUBs with an overlong token

1 participant