Base:
method
Normalize
Microsoft.ML.Tokenizers.Normalizer.Normalize(System.ReadOnlySpan<System.Char>)
14 references to Normalize
Microsoft.ML.Tokenizers (5)
Model\SentencePieceBpeModel.cs (4)
124normalizedText = text is not null ? Normalizer.Normalize(text) : Normalizer.Normalize(textSpan); 375normalizedText = Normalizer.Normalize(text); 667normalizedText = Normalizer.Normalize(textSpan); 917normalizedText = Normalizer.Normalize(textSpan);
Normalizer\SentencePieceNormalizer.cs (1)
96return Normalize(original.AsSpan());
Microsoft.ML.Tokenizers.Tests (9)
LlamaTests.cs (9)
399Assert.Equal("Hello, World!", normalizer.Normalize("Hello, World!".AsSpan())); 403Assert.Equal("Hello, World!", normalizer.Normalize("Hello, World!".AsSpan())); 407Assert.Equal(" Hello, World!", normalizer.Normalize("Hello, World!".AsSpan())); 411Assert.Equal("▁Hello,▁World!", normalizer.Normalize("Hello, World!".AsSpan())); 415Assert.Equal("▁Hello,▁▁▁▁▁▁World!", normalizer.Normalize("Hello, World!".AsSpan())); 419Assert.Equal("Hello,▁World!▁", normalizer.Normalize("Hello, World!".AsSpan())); 423Assert.Equal("Hello,▁World!", normalizer.Normalize("Hello, World!".AsSpan())); 427Assert.Equal("Hello,▁▁▁▁▁▁World!▁", normalizer.Normalize("Hello, World!".AsSpan())); 431Assert.Equal("Hello, World! ", normalizer.Normalize("Hello, World!".AsSpan()));