1 reference to PopulateIdsUpToMax
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
1385
return word.
PopulateIdsUpToMax
(accumulatedIds, maxTokens, out charsConsumed);