1 write to NormalizerSpec
Microsoft.ML.Tokenizers (1)
SentencepieceModel.cs (1)
215result.NormalizerSpec = NormalizerSpec.Parse(data, pos, pos + length);
7 references to NormalizerSpec
Microsoft.ML.Tokenizers (7)
Model\LlamaTokenizer.cs (3)
52if (modelProto.NormalizerSpec.Name != "identity" && !string.IsNullOrEmpty(modelProto.NormalizerSpec.Name)) 54throw new ArgumentException($"Normalization '{modelProto.NormalizerSpec.Name}' is not supported.", nameof(modelProto));
Model\SentencePieceBaseModel.cs (4)
33AddDummyPrefix = modelProto.NormalizerSpec.AddDummyPrefix; 34EscapeWhiteSpaces = modelProto.NormalizerSpec.EscapeWhitespaces; 55modelProto.NormalizerSpec.PrecompiledCharsmap.Span, 56modelProto.NormalizerSpec.RemoveExtraWhitespaces,