1 reference to GetPreTokenizer
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
918
IReadOnlyList<PreTokenizer>? preTokenizers =
GetPreTokenizer
(root, out bool byteLevel);