5 writes to Merges
Microsoft.ML.Tokenizers (1)
Model\BpeOptions.cs (1)
92Merges = merges;
Microsoft.ML.Tokenizers.Tests (4)
BpeTests.cs (4)
285Merges = merges?.Select(kvp => $"{kvp.Item1} {kvp.Item2}"), 405Merges = File.ReadAllLines(mergesFile).Skip(1).ToArray() // Skip the first line which is the header "#version". 927Merges = options.Merges, 990bpeOptions.Merges = GetMerges(modelElement);
3 references to Merges
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
165if (options.Merges is not null) 169foreach (string merge in options.Merges)
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
927Merges = options.Merges,