2 references to EncodeToTokens
Microsoft.ML.Tokenizers (2)
Model\WordPieceTokenizer.cs (2)
296EncodeToTokens(textSpanToEncode.Slice(split.Offset, split.Length), tokens, split.Offset); 301EncodeToTokens(textSpanToEncode, tokens, 0);