2 references to TiktokenTokenizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1366=> new TiktokenTokenizer(vocabStream, preTokenizer, specialTokens, normalizer, cacheSize); 1461return new TiktokenTokenizer(vocabStream,