1 write to _specialTokensRegex
Microsoft.ML.Tokenizers (1)
PreTokenizer\RegexPreTokenizer.cs (1)
39
_specialTokensRegex
= new Regex(string.Join("|", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled);
2 references to _specialTokensRegex
Microsoft.ML.Tokenizers (2)
PreTokenizer\RegexPreTokenizer.cs (2)
55
return SplitText(text, _regex,
_specialTokensRegex
);
106
return SplitText(buffer, _regex,
_specialTokensRegex
, text.Length);