EncodeToIds

6 references to EncodeToIds

Microsoft.ML.Tokenizers.Tests (6)

BertTokenizerTests.cs (6)

406ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 409ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false); 504ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 507ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false); 561var ids1 = bertTokenizer.EncodeToIds(text1, addSpecialTokens: false); 564var ids2 = bertTokenizer.EncodeToIds(text2, addSpecialTokens: false);