2 writes to UnknownToken
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
275
UnknownToken
= unknownToken,
898
bpeOptions.
UnknownToken
= unKnownElement.GetString();
1 reference to UnknownToken
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
186
options.
UnknownToken
,