CountTokensWithSpecialTokens

1 reference to CountTokensWithSpecialTokens

Microsoft.ML.Tokenizers (1)

Model\SentencePieceUnigramModel.cs (1)

1031CountTokensWithSpecialTokens(textToEncode, considerNormalization, ref tokenCount, buffer, ref normalizedString, out normalizedText, out charsConsumed, maxTokenCount);