2 references to TestTokenizerEncodingForTokenizer
Microsoft.ML.Tokenizers.Tests (2)
TiktokenTests.cs (2)
581TestTokenizerEncodingForTokenizer(GPT4, text, expectedTokens, expectedOffsets, expectedIds); 582TestTokenizerEncodingForTokenizer(Phi4, text, expectedTokens, expectedOffsets, expectedIds);