4 references to Phi4
Microsoft.ML.Tokenizers.Tests (4)
TiktokenTests.cs (4)
44TestGPT4TokenizationEncoding(Phi4); 582TestTokenizerEncodingForTokenizer(Phi4, text, expectedTokens, expectedOffsets, expectedIds); 755IReadOnlyList<int> encoded = Phi4.EncodeToIds(text); 757Assert.Equal(text, Phi4.Decode(encoded));