EncodeToTokens

3 references to EncodeToTokens

Microsoft.ML.Tokenizers (3)

Model\EnglishRobertaTokenizer.cs (3)

393List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping); 699List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping); 744List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping);