2 references to TiktokenTokenizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1366
=> new
TiktokenTokenizer
(vocabStream, preTokenizer, specialTokens, normalizer, cacheSize);
1461
return new
TiktokenTokenizer
(vocabStream,