2 references to Normalize
Microsoft.ML.Tokenizers (2)
Model\SentencePieceUnigramModel.cs (1)
582
int normalizationCount = Normalizer!.
Normalize
(byteSpan.Slice(0, byteCount), ref normalizationSpan, ref normalizedArrayPool);
Normalizer\SentencePieceNormalizer.cs (1)
122
int length =
Normalize
(inputArray.AsSpan(0, inputLength), ref normalized, ref poolArray);