1 reference to
Microsoft.ML.Tokenizers (1)
Model\Phi2Tokenizer.cs (1)
46
base
(vocabularyPath, mergePath, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence,