1 reference to EncodeWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeTokenizer.cs (1)
259
EncodeWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, tokens);