1 reference to CreateWhiteSpaceOrPunctuationPreTokenizer
Microsoft.ML.Tokenizers (1)
Model\BertTokenizer.cs (1)
809
PreTokenizer.
CreateWhiteSpaceOrPunctuationPreTokenizer
(splitOnSpecialTokens ? specialTokens : null) :