2 references to Replace
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
620
sb.
Replace
(EndOfWordSuffix, " ");
625
sb.
Replace
(ContinuingSubwordPrefix, string.Empty);