1 reference to EncodeToIdsWithAddedToken
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeTokenizer.cs (1)
582
EncodeToIdsWithAddedToken
(textToEncode, addBeginningOfSentence, addEndOfSentence, ids, out charsConsumed, maxTokenCount);