1 write to SpecialTokens
Microsoft.ML.Tokenizers (1)
Model\WordPieceTokenizer.cs (1)
44SpecialTokens = options.SpecialTokens;
5 references to SpecialTokens
Microsoft.ML.Tokenizers.Tests (5)
BertTokenizerTests.cs (3)
52Assert.True(tokenizer.SpecialTokens!.ContainsKey("[SPECIAL]")); 124Assert.True(tokenizer.SpecialTokens!.ContainsKey(tokenizer.UnknownToken)); 125Assert.True(tokenizer.SpecialTokens!.ContainsKey(tokenizer.ClassificationToken));
WordPieceTests.cs (2)
60Assert.Null(tokenizer.SpecialTokens); 190Assert.Equal(specialTokens, tokenizer.SpecialTokens);