1 reference to CountTokensWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeModel.cs (1)
677
CountTokensWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out charsConsumed, maxTokenCount) :