1 reference to CountIdsUpToMax
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
1031
return word.
CountIdsUpToMax
(maxTokens, out charsConsumed);