1 instantiation of CompositePreTokenizer
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
923
bpeOptions.PreTokenizer = new
CompositePreTokenizer
(preTokenizers, bpeOptions.SpecialTokens);
2 references to CompositePreTokenizer
Microsoft.ML.Tokenizers (2)
PreTokenizer\CompositePreTokenizer.cs (2)
22
/// Initializes a new instance of the <see cref="
CompositePreTokenizer
"/> class.
29
/// The <see cref="
CompositePreTokenizer
"/> can accept a list of pre-tokenizers with a maximum of 10 items.