1 reference to CountTokensFromEndWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeModel.cs (1)
929
tokenCount =
CountTokensFromEndWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out textIndex, maxTokenCount);