4 writes to UnknownToken
Microsoft.ML.Tokenizers.Tests (4)
BpeTests.cs (4)
265UnknownToken = unknownToken, 288UnknownToken = unknownToken, 896UnknownToken = "unk", 971bpeOptions.UnknownToken = unKnownElement.GetString();
2 references to UnknownToken
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
191options.UnknownToken,
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
908Assert.True(bpeTokenizer.Vocabulary.TryGetValue(options.UnknownToken, out int unkId));