2 references to TiktokenTokenizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1254
return new
TiktokenTokenizer
(
1333
return new
TiktokenTokenizer
(encoder, decoder, vocab, preTokenizer, specialTokens, normalizer, cacheSize);