2 references to EncodeToIdsFromEndResult
Microsoft.ML.Tokenizers (2)
Model\EnglishRobertaTokenizer.cs (2)
706
return
EncodeToIdsFromEndResult
(hit, accumulatedIds, maxTokens, text.Length, out textIndex);
738
return
EncodeToIdsFromEndResult
(result, accumulatedIds, maxTokens, text.Length, out textIndex);