4 writes to UnknownToken
Microsoft.ML.Tokenizers.Tests (4)
BpeTests.cs (4)
265UnknownToken = unknownToken, 288UnknownToken = unknownToken, 940UnknownToken = "unk", 1015bpeOptions.UnknownToken = unKnownElement.GetString();
2 references to UnknownToken
Microsoft.ML.Tokenizers (1)
Model\BpeTokenizer.cs (1)
191options.UnknownToken,
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
952Assert.True(bpeTokenizer.Vocabulary.TryGetValue(options.UnknownToken, out int unkId));