StoreNormalizedText

4 references to StoreNormalizedText

Microsoft.ML.Tokenizers (4)

Model\SentencePieceUnigramModel.cs (4)

515StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 905StoreNormalizedText(text, ref normalizedString!, ref normalizedStringIndex); 958StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 1339StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex);