1 write to UnknownToken
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
354UnknownToken = unknownToken;
3 references to UnknownToken
Microsoft.ML.Tokenizers (3)
Model\BPETokenizer.cs (3)
1259else if (UnknownToken is not null) 1272if (!_vocab.TryGetValue(UnknownToken, out int value)) 1281if (!_vocab.TryGetValue(UnknownToken, out int value))