1 write to _preTokenizer
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
228_preTokenizer = preTokenizer ?? PreTokenizer.CreateWordOrNonWordPreTokenizer(); // Default to WordOrNonWord pre-tokenizer
6 references to _preTokenizer
Microsoft.ML.Tokenizers (6)
Model\BPETokenizer.cs (6)
285public override PreTokenizer? PreTokenizer => _preTokenizer; 311_preTokenizer, 360_preTokenizer, 416_preTokenizer, 493_preTokenizer, 541_preTokenizer,