1 reference to SplitText
Microsoft.ML.Tokenizers (1)
PreTokenizer\RobertaPreTokenizer.cs (1)
32
return
SplitText
(text, TiktokenTokenizer.P50kBaseRegex());