2 references to GetTiktokenConfigurations
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1451(Dictionary<string, int> SpecialTokens, Regex Regex, string _, Type? __, string ___) tiktokenConfiguration = GetTiktokenConfigurations(modelName);
1491(Dictionary<string, int> SpecialTokens, Regex Regex, string _, Type? __, string ___) tiktokenConfiguration = GetTiktokenConfigurations(modelName);