1 reference to EncodeToIds
Microsoft.ML.Tokenizers (1)
Model\SentencePieceBpeModel.cs (1)
342
return
EncodeToIds
(text is null ? textSpan : text.AsSpan(), addBeginningOfSentence, addEndOfSentence, considerNormalization, out normalizedText, out charsConsumed, maxTokenCount);