2 references to EncodeToIdsFromEndResult
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (2)
1189return EncodeToIdsFromEndResult(hit, accumulatedIds, maxTokens, textSpan.Length, out textIndex); 1219return EncodeToIdsFromEndResult(result, accumulatedIds, maxTokens, textSpan.Length, out textIndex);