1 reference to LlamaTokenizer
Microsoft.ML.Tokenizers (1)
Model\LlamaTokenizer.cs (1)
57
return new
LlamaTokenizer
(modelProto, addBeginOfSentence, addEndOfSentence, specialTokens);