2 references to WordToIds
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1070
return
WordToIds
(ref hit, accumulatedIds, out charsConsumed, text.Length, maxTokens);
1085
return
WordToIds
(ref word, accumulatedIds, out charsConsumed, text.Length, maxTokens);