1 reference to SplitText
Microsoft.ML.Tokenizers (1)
PreTokenizer\RegexPreTokenizer.cs (1)
106
return
SplitText
(buffer, _regex, _specialTokensRegex, text.Length);