1 reference to PopulateIdsUpToMaxFromEnd
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
999
return word.
PopulateIdsUpToMaxFromEnd
(accumulatedIds, maxTokens, fullTextLength, out textIndex);