1 reference to CountTokensWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeTokenizer.cs (1)
922
CountTokensWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out charsConsumed, maxTokenCount) :