1 reference to GetIndexByTokenCountFromEnd
Microsoft.ML.Tokenizers.Tests (1)
UnigramTests.cs (1)
515
index = _unigramTokenizer.
GetIndexByTokenCountFromEnd
(inputText.AsSpan(), addBeginningOfSentence: false, addEndOfSentence: false, maxTokenCount: 1, considerNormalization: true, out normalized, out charConsumed);