2 references to TestDecodingWithSpan
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
313
TestDecodingWithSpan
(bpe, encodingIds, considerSpecialTokens: true, decodedTokens);
314
TestDecodingWithSpan
(bpe, encodingIds, considerSpecialTokens: false, decodedTokensWithoutUnknownToken);