1 reference to EncodeToIdsWithSpecialTokens
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
641
EncodeToIdsWithSpecialTokens
(textToEncode, considerNormalization, ids, buffer, ref normalizedString, out normalizedText, out charsConsumed, maxTokenCount);