1 reference to CountTokensFromEndWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
1278
tokenCount =
CountTokensFromEndWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out textIndex, maxTokenCount);