1 write to TreatWhitespaceAsSuffix
Microsoft.ML.Tokenizers (1)
Normalizer\SentencePieceNormalizer.cs (1)
43TreatWhitespaceAsSuffix = treatWhitespaceAsSuffix;
4 references to TreatWhitespaceAsSuffix
Microsoft.ML.Tokenizers (4)
Normalizer\SentencePieceNormalizer.cs (4)
192if (AddDummyPrefix && !TreatWhitespaceAsSuffix) 227if (AddDummyPrefix && TreatWhitespaceAsSuffix) 460if (!TreatWhitespaceAsSuffix && AddDummyPrefix) 536if (TreatWhitespaceAsSuffix && AddDummyPrefix)