2 references to Create
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
93
=>
Create
(vocabFile, mergesFile, preTokenizer: PreTokenizer.CreateWordOrNonWord(), normalizer: null, unknownToken: null, continuingSubwordPrefix: null, endOfWordSuffix: null, fuseUnknownTokens: false);
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
254
BpeTokenizer bpe = BpeTokenizer.
Create
(vocabFile: vocabFile, mergesFile: mergesFile, preTokenizer: PreTokenizer.CreateWordOrNonWord(), normalizer: null, unknownToken: unknownToken,