4 references to EncodeToIdsResult
Microsoft.ML.Tokenizers (4)
Model\TiktokenTokenizer.cs (4)
430return EncodeToIdsResult(value, accumulatedIds, maxTokenCount, text.Length, out charsConsumed);
461return EncodeToIdsResult(encodedTokens, accumulatedIds, maxTokenCount, text.Length, out charsConsumed);
596return EncodeToIdsResult(value, accumulatedIds: null, maxTokens, text.Length, out charsConsumed);
626return EncodeToIdsResult(encodedTokens, accumulatedIds: null, maxTokens, text.Length, out charsConsumed);