2 references to CreateForModel
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1522
=>
CreateForModel
(GetModelEncoding(modelName), modelName, extraSpecialTokens, normalizer);
1568
return
CreateForModel
(modelEncoding, modelName: null, extraSpecialTokens, normalizer);