1 write to SpecialTokens
Microsoft.ML.Tokenizers (1)
Model\CodeGenTokenizer.cs (1)
133
SpecialTokens
= specialTokens;
2 references to SpecialTokens
Microsoft.ML.Tokenizers.Tests (2)
CodeGenTests.cs (2)
499
var reverseSpecialTokens = codeGenTokenizer.
SpecialTokens
?.ToDictionary(kvp => kvp.Value, kvp => kvp.Key);
524
return codeGenTokenizer.
SpecialTokens
![token];