StoreNormalizedText

5 references to StoreNormalizedText

Microsoft.ML.Tokenizers (5)

Model\SentencePieceUnigramModel.cs (5)

448StoreNormalizedText(normalizationSpan, ref normalizedString, ref normalizedStringIndex); 674StoreNormalizedText(normalizationSpan, ref normalizedString!, ref normalizedStringIndex); 976StoreNormalizedText(normalizationSpan, ref normalizedString, ref normalizedStringIndex); 1202StoreNormalizedText(normalizationSpan, ref normalizedString, ref normalizedStringIndex); 1251StoreNormalizedText(normalizationSpan, ref normalizedString, ref normalizedStringIndex);