1 reference to CountTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
843
=>
CountTokens
(text is null ? textSpan : text.AsSpan(), addBeginningOfSentence, addEndOfSentence, considerPreTokenization, considerNormalization, out normalizedText, out charsConsumed, maxTokenCount);