3 references to CreateAsync
Microsoft.ML.Tokenizers (2)
Model\TiktokenTokenizer.cs (2)
1345
return await
CreateAsync
(vocabStream, preTokenizer, normalizer, specialTokens, cacheSize, cancellationToken).ConfigureAwait(false);
1419
return await
CreateAsync
(vocabStream,
Microsoft.ML.Tokenizers.Tests (1)
TiktokenTests.cs (1)
73
tokenizer = await TiktokenTokenizer.
CreateAsync
(stream, GPT4.PreTokenizer, normalizer: null, specialTokens);