1 write to _whiteSpaceOrPunctuationPreTokenizer
Microsoft.ML.Tokenizers (1)
PreTokenizer\PreTokenizer.cs (1)
68
return
_whiteSpaceOrPunctuationPreTokenizer
??= new RegexPreTokenizer(WhiteSpaceOrPunctuationRegex(), null);