1 reference to CreateHarmonyEncodingSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1221
return (
CreateHarmonyEncodingSpecialTokens
(), O200kBaseRegex(), O200kBaseFile, Type.GetType(O200kBaseTypeName), O200kBasePackageName);