ByteToUnicode

1 write to ByteToUnicode

Microsoft.ML.Tokenizers.Tests (1)

src\Microsoft.ML.Tokenizers\Utils\ByteToUnicodeEncoding.cs (1)

35ByteToUnicode = byteToUnicodeMapping;

11 references to ByteToUnicode

Microsoft.ML.Tokenizers.Tests (11)

src\Microsoft.ML.Tokenizers\Utils\ByteToUnicodeEncoding.cs (1)

36UnicodeToByte = ByteToUnicode.ToDictionary(kv => kv.Value, kv => kv.Key);

src\Microsoft.ML.Tokenizers\Utils\Helpers.cs (10)

172destination[targetIndex] = byteToUnicodeEncoder.ByteToUnicode[(char)c]; 181destination[targetIndex] = byteToUnicodeEncoder.ByteToUnicode[(char)((c + (0b110u << 11)) >> 6)]; 182destination[targetIndex + 1] = byteToUnicodeEncoder.ByteToUnicode[(char)((c & 0x3Fu) + 0x80u)]; 192destination[targetIndex] = byteToUnicodeEncoder.ByteToUnicode[(char)((value + (0b11110 << 21)) >> 18)]; 193destination[targetIndex + 1] = byteToUnicodeEncoder.ByteToUnicode[(char)(((value & (0x3Fu << 12)) >> 12) + 0x80u)]; 194destination[targetIndex + 2] = byteToUnicodeEncoder.ByteToUnicode[(char)(((value & (0x3Fu << 6)) >> 6) + 0x80u)]; 195destination[targetIndex + 3] = byteToUnicodeEncoder.ByteToUnicode[(char)((value & 0x3Fu) + 0x80u)]; 203destination[targetIndex] = byteToUnicodeEncoder.ByteToUnicode[(char)((c + (0b1110 << 16)) >> 12)]; 204destination[targetIndex + 1] = byteToUnicodeEncoder.ByteToUnicode[(char)(((c & (0x3Fu << 6)) >> 6) + 0x80u)]; 205destination[targetIndex + 2] = byteToUnicodeEncoder.ByteToUnicode[(char)((c & 0x3Fu) + 0x80u)];