2 references to EnglishRobertaTokenizer
Microsoft.ML.Tokenizers (2)
Model\EnglishRobertaTokenizer.cs (2)
76
=> new
EnglishRobertaTokenizer
(vocabularyStream, mergeStream, highestOccurrenceMappingStream);
95
=> new
EnglishRobertaTokenizer
(vocabularyStream, mergeStream, highestOccurrenceMappingStream, preTokenizer, normalizer, filterUnsupportedChars);