1 write to _specialTokensReverse
Microsoft.ML.Tokenizers (1)
Model\SentencePieceTokenizer.cs (1)
81_specialTokensReverse = new Dictionary<int, string>();
4 references to _specialTokensReverse
Microsoft.ML.Tokenizers (4)
Model\SentencePieceTokenizer.cs (4)
86_specialTokensReverse.Add(item.Value, item.Key); 290tokens.Add(new EncodedToken(id, _specialTokensReverse![id], new Range(Offset, Offset + Length))); 1758else if (tokenizer._specialTokensReverse?.TryGetValue(id, out string? specialToken) is true) 1999else if (!tokenizer._specialTokensReverse?.TryGetValue(id, out specialToken) is true)