1 reference to BertNormalizer
Microsoft.ML.Tokenizers (1)
Model\BertTokenizer.cs (1)
763
options.Normalizer ??= options.ApplyBasicTokenization ? new
BertNormalizer
(options.LowerCaseBeforeTokenization, options.IndividuallyTokenizeCjk, options.RemoveNonSpacingMarks) : null;