2 writes to _preTokenizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
93_preTokenizer = preTokenizer; 109_preTokenizer = preTokenizer;
5 references to _preTokenizer
Microsoft.ML.Tokenizers (5)
Model\TiktokenTokenizer.cs (5)
127public override PreTokenizer? PreTokenizer => _preTokenizer; 271_preTokenizer, 381_preTokenizer, 550_normalizer, _preTokenizer, 677_preTokenizer,