1 reference to EncodeNextUtf8
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
478
int codepointLength = Helpers.
EncodeNextUtf8
(normalizationSpan.Slice(offsetStart), destination);