neural-tokenizers

Repository contains a python package for using segmentation methods built on top of (small) transformer language models.