EncodeToIdsFromEndWithCache

2 references to EncodeToIdsFromEndWithCache

Microsoft.ML.Tokenizers (2)

Model\BPETokenizer.cs (2)

717tokenCount += EncodeToIdsFromEndWithCache(textSpanToEncode.Slice(split.Offset, split.Length), null, maxTokenCount - tokenCount, out int textIndex, ref priorityQueue); 728tokenCount = EncodeToIdsFromEndWithCache(textSpanToEncode, null, maxTokenCount - (EndOfSentenceToken is null ? 0 : 1), out int charsConsumed, ref priorityQueue);