3 references to CreateWhiteSpacePreTokenizer
Microsoft.ML.Tokenizers (2)
Model\BertTokenizer.cs (1)
810
PreTokenizer.
CreateWhiteSpacePreTokenizer
();
Model\WordPieceTokenizer.cs (1)
77
_preTokenizer = preTokenizer ?? PreTokenizer.
CreateWhiteSpacePreTokenizer
(specialTokens);
Microsoft.ML.Tokenizers.Tests (1)
PreTokenizerTests.cs (1)
35
PreTokenizer.
CreateWhiteSpacePreTokenizer
(),