EncodeToIdsInternal

4 references to EncodeToIdsInternal

Microsoft.ML.Tokenizers (4)

Model\EnglishRobertaTokenizer.cs (4)

441EncodeToIdsInternal(textSpanToEncode.Slice(split.Offset, split.Length), ids, out int length, maxTokenCount - ids.Count); 452EncodeToIdsInternal(textSpanToEncode, ids, out textLength, maxTokenCount); 525count += EncodeToIdsInternal(textSpanToEncode.Slice(split.Offset, split.Length), null, out int length, maxTokenCount - count); 536count += EncodeToIdsInternal(textSpanToEncode, null, out charsConsumed, maxTokenCount);