1 reference to BertNormalizer
Microsoft.ML.Tokenizers (1)
Model\BertTokenizer.cs (1)
763options.Normalizer ??= options.ApplyBasicTokenization ? new BertNormalizer(options.LowerCaseBeforeTokenization, options.IndividuallyTokenizeCjk, options.RemoveNonSpacingMarks) : null;