1 write to _preTokenizer
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
311_preTokenizer = preTokenizer ?? PreTokenizer.CreateWordOrNonWord(); // Default to WordOrNonWord pre-tokenizer
6 references to _preTokenizer
Microsoft.ML.Tokenizers (6)
Model\BPETokenizer.cs (6)
418public override PreTokenizer? PreTokenizer => _preTokenizer; 444_preTokenizer, 503_preTokenizer, 570_preTokenizer, 652_preTokenizer, 705_preTokenizer,