EncodeAsBytesFromEnd

2 references to EncodeAsBytesFromEnd

Microsoft.ML.Tokenizers (2)

Model\SentencePieceTokenizer.cs (2)

1401if (!EncodeAsBytesFromEnd(text.Slice(symbols[index].pieceSpan.Index, symbols[index].pieceSpan.Length), symbols[index].pieceSpan.Index, ref textIndex)) 1500return EncodeAsBytesFromEnd(text.Slice(pieceSpan.Index, pieceSpan.Length), pieceSpan.Index, ref textIndex);