1 write to ByteLevel
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
992bpeOptions.ByteLevel = byteLevel;
2 references to ByteLevel
Microsoft.ML.Tokenizers (2)
Model\BpeOptions.cs (1)
121/// which discards whitespace regardless of the <see cref="ByteLevel"/> setting. Each segment is encoded
Model\BPETokenizer.cs (1)
195options.ByteLevel,