6 references to EncodeToUtf8AndTransform
Microsoft.ML.Tokenizers (6)
Model\BpeTokenizer.cs (3)
1354int encodedLength = Helpers.EncodeToUtf8AndTransform(text, token, mapping); 1467int encodedLength = Helpers.EncodeToUtf8AndTransform(text, token, mapping); 1547int encodedLength = Helpers.EncodeToUtf8AndTransform(text, token, mapping);
Model\CodeGenTokenizer.cs (3)
462int encodedLength = Helpers.EncodeToUtf8AndTransform(textSpan, token, mapping); 1145int encodedLength = Helpers.EncodeToUtf8AndTransform(textSpan, token, mapping); 1209int encodedLength = Helpers.EncodeToUtf8AndTransform(textSpan, token, mapping);