EncodeToIdsFromEndResult

2 references to EncodeToIdsFromEndResult

Microsoft.ML.Tokenizers (2)

Model\TiktokenTokenizer.cs (2)

721return EncodeToIdsFromEndResult(value, accumulatedIds: null, maxTokens, text.Length, out textIndex); 751return EncodeToIdsFromEndResult(encodedTokens, accumulatedIds: null, maxTokens, text.Length, out textIndex);