3 references to Create
Microsoft.ML.GenAI.Phi (1)
Phi2\Phi2TokenizerHelper.cs (1)
28
return CodeGenTokenizer.
Create
(vocabStream, mergesStream, addPrefixSpace, addBeginOfSentence, addEndOfSentence);
Microsoft.ML.Tokenizers.Tests (2)
CodeGenTests.cs (2)
31
return CodeGenTokenizer.
Create
(vocabStream, mergesStream, addPrefixSpace, bos, eos);
42
return CodeGenTokenizer.
Create
(vocabStream, mergesStream);