EncodeToTokens

3 references to EncodeToTokens

Microsoft.ML.Tokenizers (3)

Model\EnglishRobertaTokenizer.cs (3)

382List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping); 688List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping); 733List<EncodedToken> result = EncodeToTokens(token.AsSpan().Slice(0, newTokenIndex), indexMapping);