1 reference to CompositePreTokenizer
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
923
bpeOptions.PreTokenizer = new
CompositePreTokenizer
(preTokenizers, bpeOptions.SpecialTokens);