1 reference to CountTokensWithoutSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
1035
CountTokensWithoutSpecialTokens
(textToEncode, considerNormalization, ref tokenCount, buffer, ref normalizedString, out normalizedText, out charsConsumed, maxTokenCount);