1 write to UnicodeToByte
Microsoft.ML.Tokenizers (1)
Utils\ByteToUnicodeEncoding.cs (1)
36UnicodeToByte = ByteToUnicode.ToDictionary(kv => kv.Value, kv => kv.Key);
3 references to UnicodeToByte
Microsoft.ML.Tokenizers (3)
Model\CodeGenTokenizer.cs (2)
1476current[i] = (byte)byteToUnicodeEncoding.UnicodeToByte[span[i]]; 1566IReadOnlyDictionary<char, char> unicodeToByte = ByteToUnicodeEncoding.Instance.UnicodeToByte;
Model\EnglishRobertaTokenizer.cs (1)
286IReadOnlyDictionary<char, char> unicodeToByte = ByteToUnicodeEncoding.Instance.UnicodeToByte;