1 reference to GetIndexByTokenCountFromEnd
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
1163
return
GetIndexByTokenCountFromEnd
(text, textSpan, settings.MaxTokenCount, settings.ConsiderNormalization, out normalizedString, out tokenCount);