GetIndexByTokenCountFromEndWithSpecialTokens

1 reference to GetIndexByTokenCountFromEndWithSpecialTokens

Microsoft.ML.Tokenizers (1)

Model\SentencePieceUnigramModel.cs (1)

1300GetIndexByTokenCountFromEndWithSpecialTokens(textToEncode, considerNormalization, ref tokenCount, buffer, ref normalizedString, out charConsumedFromEnd, out normalizedText, maxTokenCount);