1 reference to IMSep
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1136
{ IMEnd, 100265 }, {
IMSep
, 100266 }, { "<|dummy_85|>", 100349}, // <|dummy_85|> is used for padding according to the phi-4 special token mapping.