1 write to _mergeRanks
Microsoft.ML.Tokenizers (1)
Model\CodeGenTokenizer.cs (1)
126_mergeRanks = GetMergeRanks(mergeStream);
1 reference to _mergeRanks
Microsoft.ML.Tokenizers (1)
Model\CodeGenTokenizer.cs (1)
1711if (!_mergeRanks.TryGetValue(textSpan.Slice(symbols[left].pieceSpan.Index, symbols[left].pieceSpan.Length), textSpan.Slice(symbols[right].pieceSpan.Index, symbols[right].pieceSpan.Length), out int rank))