2 writes to _normalizer
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
93_normalizer = normalizer; 109_normalizer = normalizer;
5 references to _normalizer
Microsoft.ML.Tokenizers (5)
Model\TiktokenTokenizer.cs (5)
131public override Normalizer? Normalizer => _normalizer; 269_normalizer, 379_normalizer, 549_normalizer, _preTokenizer, 675_normalizer,