1 write to NumAttentionHeads
Microsoft.ML.GenAI.Mistral (1)
MistralConfig.cs (1)
24this.NumAttentionHeads = 32;
4 references to NumAttentionHeads
Microsoft.ML.GenAI.Mistral (4)
MistralConfig.cs (1)
34this.HeadDim = this.HiddenSize / this.NumAttentionHeads;
MistralDecoderLayer.cs (3)
91var headDim = config.HiddenSize / config.NumAttentionHeads; 95numHeads: config.NumAttentionHeads, 98numKeyValueGroups: config.NumAttentionHeads / config.NumKeyValueHeads,