2 references to SplitText
Microsoft.ML.Tokenizers (2)
PreTokenizer\PreTokenizer.cs (1)
144
return
SplitText
(text.ToString(), regex);
PreTokenizer\RobertaPreTokenizer.cs (1)
32
return
SplitText
(text, TiktokenTokenizer.P50kBaseRegex());