1 reference to EncodeToTokensWithoutSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
173
EncodeToTokensWithoutSpecialTokens
(textToEncode, addBeginningOfSentence, addEndOfSentence, considerNormalization, tokens, buffer, ref normalizedString, out normalizedText);