1 reference to CountTokens
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
484
tokenCount =
CountTokens
(text, textSpan, settings.ConsiderPreTokenization, settings.ConsiderNormalization, out normalizedText, out int charsConsumed, settings.MaxTokenCount);