EncodeInternal

3 references to EncodeInternal

Microsoft.ML.Tokenizers (3)

Model\SentencePieceBpeModel.cs (3)

99EncodeInternal(textToEncode, addBeginningOfSentence, addEndOfSentence, tokens); 120EncodeInternal(text.Slice(currentOffset, Offset - currentOffset), addBeginOfSentence: false, addEndOfSentence: false, tokens); 133EncodeInternal(text.Slice(currentOffset), addBeginOfSentence: false, addEndOfSentence: false, tokens);