1 reference to PreTokenize
Microsoft.ML.Tokenizers (1)
PreTokenizer\CompositePreTokenizer.cs (1)
152
IEnumerable<(int Offset, int Length)> result =
PreTokenize
(buffer, text.Length);