2 references to Create
Microsoft.ML.GenAI.LLaMA (1)
LlamaTokenizerHelper.cs (1)
53
return TiktokenTokenizer.
Create
(File.OpenRead(modelFilePath), preTokenizer, normalizer: null, specialTokens: _specialTokens);
Microsoft.ML.Tokenizers.Tests (1)
TiktokenTests.cs (1)
64
tokenizer = TiktokenTokenizer.
Create
(stream, GPT4.PreTokenizer, null, specialTokensEncoder);