2 references to TestTokenizerEncodingForTokenizer
Microsoft.ML.Tokenizers.Tests (2)
TiktokenTests.cs (2)
608
TestTokenizerEncodingForTokenizer
(GPT4, text, expectedTokens, expectedOffsets, expectedIds);
609
TestTokenizerEncodingForTokenizer
(Phi4, text, expectedTokens, expectedOffsets, expectedIds);