StoreNormalizedText

4 references to StoreNormalizedText

Microsoft.ML.Tokenizers (4)

Model\SentencePieceUnigramModel.cs (4)

290StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 669StoreNormalizedText(text, ref normalizedString!, ref normalizedStringIndex); 722StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex); 1093StoreNormalizedText(text.Slice(Offset, Length), ref normalizedString, ref normalizedStringIndex);