1 reference to CountTokensWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
923
CountTokensWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out charsConsumed, maxTokenCount) :