2 references to EnglishRobertaTokenizer
Microsoft.ML.Tokenizers (2)
Model\EnglishRobertaTokenizer.cs (2)
85
=> new
EnglishRobertaTokenizer
(vocabularyStream, mergeStream, highestOccurrenceMappingStream);
107
=> new
EnglishRobertaTokenizer
(vocabularyStream, mergeStream, highestOccurrenceMappingStream, preTokenizer, normalizer, filterUnsupportedChars);