1 reference to Create
Microsoft.ML.Tokenizers.Tests (1)
TiktokenTests.cs (1)
76
tokenizer = TiktokenTokenizer.
Create
(stream, GPT4.PreTokenizer, null, specialTokens);