1 write to Normalizer
Microsoft.ML.Tokenizers (1)
Model\BertTokenizer.cs (1)
763
options.
Normalizer
??= options.ApplyBasicTokenization ? new BertNormalizer(options.LowerCaseBeforeTokenization, options.IndividuallyTokenizeCjk, options.RemoveNonSpacingMarks) : null;
2 references to Normalizer
Microsoft.ML.Tokenizers (2)
Model\WordPieceOptions.cs (1)
24
/// Gets or sets the <see cref="
Normalizer
"/> to override the default normalizer, if desired.
Model\WordPieceTokenizer.cs (1)
73
_normalizer = options.
Normalizer
;