2 references to CodeGenTokenizer
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (1)
1894
return new
CodeGenTokenizer
(
Model\Phi2Tokenizer.cs (1)
77
base
(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence,