EncodeToIds

6 references to EncodeToIds

Microsoft.ML.Tokenizers.Tests (6)

BertTokenizerTests.cs (6)

316ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 319ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false); 414ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 417ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false); 471var ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 474var ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false);