2 references to EncodeToTokens
Microsoft.ML.Tokenizers (2)
Model\WordPieceTokenizer.cs (2)
351EncodeToTokens(textSpanToEncode.Slice(split.Offset, split.Length), tokens, split.Offset); 356EncodeToTokens(textSpanToEncode, tokens, 0);