decode raises Tiktoken::UnicodeError when tokens are truncated mid multi-byte character (diverges from Python tiktoken)

[yvonnenieuwerth]

Summary

Tiktoken::Encoding#decode raises Tiktoken::UnicodeError when the token array ends in the middle of a multi-byte UTF-8 character. BPE tokens are byte-level, so a single character (emoji, many non-Latin scripts) can span multiple tokens — truncating a token array to a limit can therefore produce a prefix that is not valid UTF-8.

Python's reference implementation handles this gracefully: tiktoken's decode uses errors="replace" by default, so it returns a string with � instead of raising. tiktoken_ruby raises instead, which makes the very common "truncate text to N tokens" pattern fragile:

tokens = enc.encode(text)
enc.decode(tokens.first(max_tokens))  # 💥 sometimes

Reproduction

require "tiktoken_ruby"

enc = Tiktoken.encoding_for_model("gpt-4o")

tokens = enc.encode("🦄")  # => 3 tokens (the emoji is 4 bytes, split across tokens)
enc.decode(tokens.first(2))
# => Tiktoken::UnicodeError: Unable to decode into a valid UTF-8 string:
#    incomplete utf-8 byte sequence from index 0

Same for e.g. "थ" (2 tokens in cl100k_base) or "𓀀" (4 tokens). Python equivalent for comparison:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
enc.decode(enc.encode("🦄")[:2])  # => '�' (no error)

Tested with tiktoken_ruby 0.0.15.1 on Ruby (arm64-darwin and aarch64-linux).

Cause

CoreBPEWrapper#decode calls tiktoken-rs's CoreBPE::decode, which does a strict UTF-8 conversion and returns an error for invalid byte sequences; the wrapper converts that into Tiktoken::UnicodeError.

Suggestion

One or more of:

1. Match Python tiktoken's default and decode lossily (String::from_utf8_lossy), or
2. Add an opt-in, e.g. decode(tokens, errors: :replace), keeping the raising behaviour as default, or
3. Expose decode_bytes(tokens) returning a binary string so callers can handle invalid sequences themselves (this also matches an API Python tiktoken has).