2 references to WordToIdsFromEnd
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1535int res = WordToIdsFromEnd(ref hit, accumulatedIds, out textIndex, textSpan.Length, maxTokens); 1557int result = WordToIdsFromEnd(ref word, accumulatedIds, out textIndex, textSpan.Length, maxTokens);