2 references to SimpleWithUnknownTokenTest
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
259SimpleWithUnknownTokenTest(bpe, sentence, offsets, ids, expectedTokens, decodedTokens, decodedTokensWithoutUnknownToken);
282SimpleWithUnknownTokenTest(bpe1, sentence, offsets, ids, expectedTokens, decodedTokens, decodedTokensWithoutUnknownToken);