2 references to Bpe
Microsoft.ML.Tokenizers (2)
Model\LlamaTokenizer.cs (1)
47
if (modelProto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.
Bpe
)
Model\SentencePieceTokenizer.cs (1)
27
TrainerSpec.Types.ModelType.
Bpe
=> new SentencePieceBpeModel(modelProto, addBos, addEos, specialTokens),