1 reference to EncodeToTokensWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
169
EncodeToTokensWithSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, considerNormalization, tokens, buffer, ref normalizedString, out normalizedText);