Skip to content

Support any LLMLingua-2-style token classifier (detect subword style from the tokenizer) - #261

Open
Dario A (riceharvest) wants to merge 1 commit into
microsoft:mainfrom
riceharvest:subword-style-detection
Open

Support any LLMLingua-2-style token classifier (detect subword style from the tokenizer)#261
Dario A (riceharvest) wants to merge 1 commit into
microsoft:mainfrom
riceharvest:subword-style-detection

Commits

  1. Commits on Oct 8, 2026