1 reference to IsCjkChar
Microsoft.ML.Tokenizers (1)
Normalizer\BertNormalizer.cs (1)
94
if (_individuallyTokenizeCjk &&
IsCjkChar
(codePoint))