1 write to ByteLevel
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
1036bpeOptions.ByteLevel = byteLevel;
2 references to ByteLevel
Microsoft.ML.Tokenizers (2)
Model\BpeOptions.cs (1)
104/// which discards whitespace regardless of the <see cref="ByteLevel"/> setting. Each segment is encoded
Model\BpeTokenizer.cs (1)
195options.ByteLevel,