1 reference to TiktokenTokenizer
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1261
=> new
TiktokenTokenizer
(vocabFilePath, preTokenizer, specialTokens, normalizer, cacheSize);