3 references to CreateWhiteSpace
Microsoft.ML.Tokenizers (2)
Model\BertTokenizer.cs (1)
809
options.PreTokenizer ??= options.ApplyBasicTokenization ? PreTokenizer.CreateWordOrPunctuation(options.SplitOnSpecialTokens ? specialTokensDict : null) : PreTokenizer.
CreateWhiteSpace
();
Model\WordPieceTokenizer.cs (1)
72
_preTokenizer = options.PreTokenizer ?? PreTokenizer.
CreateWhiteSpace
(options.SpecialTokens);
Microsoft.ML.Tokenizers.Tests (1)
PreTokenizerTests.cs (1)
35
PreTokenizer.
CreateWhiteSpace
(),