2 references to WordToIdsFromEnd
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1103
return
WordToIdsFromEnd
(ref hit, accumulatedIds, out textIndex, text.Length, maxTokens);
1118
return
WordToIdsFromEnd
(ref word, accumulatedIds, out textIndex, text.Length, maxTokens);