1 reference to CreateAsync
Microsoft.ML.Tokenizers.Tests (1)
TiktokenTests.cs (1)
68
tokenizer = await TiktokenTokenizer.
CreateAsync
(tokenizerDataFileName, GPT4.PreTokenizer, normalizer: null, specialTokens);