2 references to AddAffixToken
Microsoft.ML.Tokenizers (2)
Model\SentencePieceTokenizer.cs (2)
815AddAffixToken(prefixTokens, "<s>", vocab, specialTokens, required: false); 816AddAffixToken(suffixTokens, "</s>", vocab, specialTokens, required: false);