EncodeToIdsWithoutSpecialTokens

1 reference to EncodeToIdsWithoutSpecialTokens

Microsoft.ML.Tokenizers (1)

Model\SentencePieceUnigramModel.cs (1)

645EncodeToIdsWithoutSpecialTokens(textToEncode, considerNormalization, ids, buffer, ref normalizedString, out normalizedText, out charsConsumed, maxTokenCount);