StoreNormalizedText

4 references to StoreNormalizedText

Microsoft.ML.Tokenizers (4)

Model\SentencePieceUnigramModel.cs (4)

240StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 619StoreNormalizedText(text, ref normalizedString!, ref normalizedStringIndex); 672StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 1043StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex);