Skip to content

Support any LLMLingua-2-style token classifier (detect subword style from the tokenizer) - #261

Open
Dario A (riceharvest) wants to merge 1 commit into
microsoft:mainfrom
riceharvest:subword-style-detection
Open

Support any LLMLingua-2-style token classifier (detect subword style from the tokenizer)#261
Dario A (riceharvest) wants to merge 1 commit into
microsoft:mainfrom
riceharvest:subword-style-detection

Detect subword style from the tokenizer so any LLMLingua-2-style clas…

1f2b881
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar