1 reference to ToTokens
Microsoft.ML.Tokenizers (1)
Model\BpeTokenizer.cs (1)
1321
internal void WordToTokens(ref Word word, List<EncodedToken> tokens, int offset, ReadOnlySpan<int> mapping) => word.
ToTokens
(VocabReverse, tokens, offset, mapping);