8 references to Instance
Microsoft.ML.Tokenizers (3)
Model\BPETokenizer.cs (3)
83=> Create(vocabFile, mergesFile, preTokenizer: WhiteSpacePreTokenizer.Instance, normalizer: null, unknownToken: null, continuingSubwordPrefix: null, endOfWordSuffix: null, fuseUnknownTokens: false); 125=> Create(vocabStream, mergesStream, preTokenizer: WhiteSpacePreTokenizer.Instance, normalizer: null, unknownToken: null, continuingSubwordPrefix: null, endOfWordSuffix: null, fuseUnknownTokens: false); 205_preTokenizer = preTokenizer ?? WhiteSpacePreTokenizer.Instance; // Default to WhiteSpace pre-tokenizer
Microsoft.ML.Tokenizers.Tests (5)
BpeTests.cs (2)
254BpeTokenizer bpe = BpeTokenizer.Create(vocabFile: vocabFile, mergesFile: mergesFile, preTokenizer: WhiteSpacePreTokenizer.Instance, normalizer: null, unknownToken: unknownToken, 503vocabStream: emptyVocabStream, mergesStream: null, preTokenizer: preTokenizer ?? WhiteSpacePreTokenizer.Instance, normalizer: normalizer, unknownToken: "Ukn");
PreTokenizerTests.cs (3)
21WhiteSpacePreTokenizer.Instance, 28WhiteSpacePreTokenizer.Instance, 66Assert.Empty(WhiteSpacePreTokenizer.Instance.PreTokenize((string)null!));