1 reference to CountTokensFromEndWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeTokenizer.cs (1)
1277
tokenCount =
CountTokensFromEndWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, out textIndex, maxTokenCount);