GetTiktokenConfigurations

2 references to GetTiktokenConfigurations

Microsoft.ML.Tokenizers (2)

Model\TiktokenTokenizer.cs (2)

1455(Dictionary<string, int> SpecialTokens, Regex Regex, string _, Type? __, string ___) tiktokenConfiguration = GetTiktokenConfigurations(modelName); 1495(Dictionary<string, int> SpecialTokens, Regex Regex, string _, Type? __, string ___) tiktokenConfiguration = GetTiktokenConfigurations(modelName);