2 references to TestTokenizerEncodingForTokenizer
Microsoft.ML.Tokenizers.Tests (2)
TiktokenTests.cs (2)
576
TestTokenizerEncodingForTokenizer
(GPT4, text, expectedTokens, expectedOffsets, expectedIds);
577
TestTokenizerEncodingForTokenizer
(Phi4, text, expectedTokens, expectedOffsets, expectedIds);