2 references to WordToIds
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1455int res = WordToIds(ref hit, accumulatedIds, out charsConsumed, textSpan.Length, maxTokens); 1476int result = WordToIds(ref word, accumulatedIds, out charsConsumed, textSpan.Length, maxTokens);