2 references to TestDecodingWithSpan
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
269TestDecodingWithSpan(bpe, encodingIds, considerSpecialTokens: true, decodedTokens); 270TestDecodingWithSpan(bpe, encodingIds, considerSpecialTokens: false, decodedTokensWithoutUnknownToken);