2 references to WordToTokens
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1348WordToTokens(ref word, tokens, offset, mapping); 1364WordToTokens(ref word, tokens, offset, mapping);