1 write to _specialTokensRegex
Microsoft.ML.Tokenizers (1)
PreTokenizer\RegexPreTokenizer.cs (1)
39
_specialTokensRegex
= new Regex(string.Join("|", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled);
1 reference to _specialTokensRegex
Microsoft.ML.Tokenizers (1)
PreTokenizer\RegexPreTokenizer.cs (1)
55
return SplitText(text, _regex,
_specialTokensRegex
);