1 write to NormalizationChain
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
754model.Normalizer!.NormalizationChain = chainNormalizer;
4 references to NormalizationChain
Microsoft.ML.Tokenizers (4)
Normalizer\SentencePieceNormalizer.cs (4)
142if (NormalizationChain is not null) 146return NormalizeCore(NormalizationChain.Normalize(original.ToString()).AsSpan()); 390if (NormalizationChain is not null) 395string chained = NormalizationChain.Normalize(Helpers.GetString(input));