2 references to TestTokenizerEncodingForTokenizer
Microsoft.ML.Tokenizers.Tests (2)
TiktokenTests.cs (2)
632
TestTokenizerEncodingForTokenizer
(GPT4, text, expectedTokens, expectedOffsets, expectedIds);
633
TestTokenizerEncodingForTokenizer
(Phi4, text, expectedTokens, expectedOffsets, expectedIds);