1 write to SpecialTokens
Microsoft.ML.Tokenizers (1)
Model\CodeGenTokenizer.cs (1)
133SpecialTokens = specialTokens;
2 references to SpecialTokens
Microsoft.ML.Tokenizers.Tests (2)
CodeGenTests.cs (2)
499var reverseSpecialTokens = codeGenTokenizer.SpecialTokens?.ToDictionary(kvp => kvp.Value, kvp => kvp.Key); 524return codeGenTokenizer.SpecialTokens![token];