1 write to _preTokenizer
Microsoft.ML.Tokenizers (1)
Model\WordPieceTokenizer.cs (1)
72_preTokenizer = options.PreTokenizer ?? PreTokenizer.CreateWhiteSpace(options.SpecialTokens);
5 references to _preTokenizer
Microsoft.ML.Tokenizers (5)
Model\WordPieceTokenizer.cs (5)
243public override PreTokenizer? PreTokenizer => _preTokenizer; 285_preTokenizer, 408_preTokenizer, 560_preTokenizer, 622_preTokenizer,