2 references to EncodeToTokens
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
281EncodeToTokens(textSpanToEncode.Slice(split.Offset, split.Length), tokens, split.Offset); 286EncodeToTokens(textSpanToEncode, tokens, 0);