GetEncoder

5 references to GetEncoder

Microsoft.ML.Tokenizers.Tests (5)

TiktokenTests.cs (5)

122IReadOnlyDictionary<ReadOnlyMemory<byte>, int> encoder = GetEncoder(tiktoken)!; 123IReadOnlyDictionary<ReadOnlyMemory<byte>, int> externalEncoder = GetEncoder(externalTokenizer)!; 490Assert.Equal(GetEncoder(tiktoken1), GetEncoder(tiktoken)); 531int entriesCount = GetEncoder((tokenizer as TiktokenTokenizer)!)!.Count;