2 writes to UnknownToken
Microsoft.ML.Tokenizers (2)
Model\SentencePieceBaseModel.cs (2)
31UnknownToken = modelProto.TrainerSpec.UnkPiece ?? "<unk>"; 93UnknownToken = options.UnknownToken;
3 references to UnknownToken
Microsoft.ML.Tokenizers (3)
Model\SentencePieceBaseModel.cs (2)
413token = model.UnknownToken; 651specialToken = model.UnknownToken;
Model\SentencePieceTokenizer.cs (1)
91public string UnknownToken => _model.UnknownToken;