3 writes to PreTokenizer
Microsoft.ML.Tokenizers.Tests (3)
BpeTests.cs (3)
263PreTokenizer = PreTokenizer.CreateWordOrNonWord(), 286PreTokenizer = PreTokenizer.CreateWordOrNonWord(), 996bpeOptions.PreTokenizer = new CompositePreTokenizer(preTokenizers, bpeOptions.SpecialTokens);
1 reference to PreTokenizer
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
188options.PreTokenizer,