1 reference to CountTokensFromEndWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
1277
tokenCount =
CountTokensFromEndWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out textIndex, maxTokenCount);