1 reference to PopulateIdsUpToMax
Microsoft.ML.Tokenizers (1)
Model\BpeTokenizer.cs (1)
1405return word.PopulateIdsUpToMax(accumulatedIds, maxTokens, out charsConsumed);