2 references to WordToTokens
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
1348
WordToTokens
(ref word, tokens, offset, mapping);
1364
WordToTokens
(ref word, tokens, offset, mapping);