3 references to CreateAsync
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1427
return await
CreateAsync
(vocabStream, preTokenizer, normalizer, specialTokens, cacheSize, cancellationToken).ConfigureAwait(false);
1501
return await
CreateAsync
(vocabStream,
Microsoft.ML.Tokenizers.Tests (1)
TiktokenTests.cs (1)
78
tokenizer = await TiktokenTokenizer.
CreateAsync
(stream, GPT4.PreTokenizer, normalizer: null, specialTokens);