2 references to EncodeToIds
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
393EncodeToIds(textSpanToEncode.Slice(split.Offset, split.Length), ids, out int length, maxTokenCount - ids.Count); 404EncodeToIds(textSpanToEncode, ids, out charsConsumed);