Base:
method
PreTokenize
Microsoft.ML.Tokenizers.PreTokenizer.PreTokenize(System.ReadOnlySpan<System.Char>)
1 reference to PreTokenize
Microsoft.ML.Tokenizers.Tests (1)
PreTokenizerTests.cs (1)
123return PreTokenize(text.AsSpan());