2 references to CodeGenTokenizer
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (1)
1894return new CodeGenTokenizer(
Model\Phi2Tokenizer.cs (1)
77base(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence,