GetIndexByTokenCountFromEndWithoutSpecialTokens

1 reference to GetIndexByTokenCountFromEndWithoutSpecialTokens

Microsoft.ML.Tokenizers (1)

Model\SentencePieceUnigramModel.cs (1)

1304GetIndexByTokenCountFromEndWithoutSpecialTokens(textToEncode, considerNormalization, ref tokenCount, buffer, ref normalizedString, out charConsumedFromEnd, out normalizedText, maxTokenCount);