2 references to AddData
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
200AddData(Convert.FromBase64String(line), lineNumber); 218AddData(Helpers.FromBase64String(line, 0, spaceIndex), rank);