1 reference to GetIndexByTokenCountFromEnd
Microsoft.ML.Tokenizers.Tests (1)
LlamaTests.cs (1)
699
index = tokenizer.
GetIndexByTokenCountFromEnd
(s, addBeginningOfSentence: false, addEndOfSentence: false, maxTokenCount: i, considerNormalization: true, out normalizedString, out tokenCount);