1 reference to EncodeNextUtf8
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
428
int codepointLength = Helpers.
EncodeNextUtf8
(normalizationSpan.Slice(offsetStart), destination);