2 references to EncodeToTokens
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
282EncodeToTokens(textSpanToEncode.Slice(split.Offset, split.Length), tokens, split.Offset); 287EncodeToTokens(textSpanToEncode, tokens, 0);