Package Usage: pypi: curated-tokenizers
Lightweight piece tokenization library
10 versions
Latest release: over 1 year ago
2 dependent packages
107,769 downloads last month
View more package details: https://packages.ecosystem.code.gouv.fr/registries/pypi.org/packages/curated-tokenizers