1 reference to GetIndexByTokenCount
Microsoft.ML.Tokenizers.Tests (1)
LlamaTests.cs (1)
691
int index = tokenizer.
GetIndexByTokenCount
(s, addBeginningOfSentence: false, addEndOfSentence: false, maxTokenCount: i, out normalizedString, out tokenCount);