1 reference to LastIndexOf
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
650
return
LastIndexOf
(text, textSpan, settings.MaxTokenCount, settings.ConsiderNormalization, settings.ConsiderNormalization, out normalizedText, out tokenCount);