2 writes to _normalizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
94_normalizer = normalizer; 110_normalizer = normalizer;
5 references to _normalizer
Microsoft.ML.Tokenizers (5)
Model\TiktokenTokenizer.cs (5)
132public override Normalizer? Normalizer => _normalizer; 270_normalizer, 380_normalizer, 550_normalizer, _preTokenizer, 676_normalizer,