2 references to WhiteSpaceOrPunctuationRegex
Microsoft.ML.Tokenizers (2)
PreTokenizer\PreTokenizer.cs (2)
68return _whiteSpaceOrPunctuationPreTokenizer ??= new RegexPreTokenizer(WhiteSpaceOrPunctuationRegex(), null); 71return new RegexPreTokenizer(WhiteSpaceOrPunctuationRegex(), specialTokens);