1 write to UnicodeToByte
Microsoft.ML.Tokenizers (1)
Utils\ByteToUnicodeEncoding.cs (1)
36
UnicodeToByte
= ByteToUnicode.ToDictionary(kv => kv.Value, kv => kv.Key);
3 references to UnicodeToByte
Microsoft.ML.Tokenizers (3)
Model\CodeGenTokenizer.cs (2)
1476
current[i] = (byte)byteToUnicodeEncoding.
UnicodeToByte
[span[i]];
1566
IReadOnlyDictionary<char, char> unicodeToByte = ByteToUnicodeEncoding.Instance.
UnicodeToByte
;
Model\EnglishRobertaTokenizer.cs (1)
286
IReadOnlyDictionary<char, char> unicodeToByte = ByteToUnicodeEncoding.Instance.
UnicodeToByte
;