1 write to TreatWhitespaceAsSuffix
Microsoft.ML.Tokenizers (1)
Normalizer\SentencePieceNormalizer.cs (1)
27
TreatWhitespaceAsSuffix
= treatWhitespaceAsSuffix;
2 references to TreatWhitespaceAsSuffix
Microsoft.ML.Tokenizers (2)
Normalizer\SentencePieceNormalizer.cs (2)
116
if (AddDummyPrefix && !
TreatWhitespaceAsSuffix
)
151
if (AddDummyPrefix &&
TreatWhitespaceAsSuffix
)