3 references to SegmentFromEnd
Microsoft.ML.Tokenizers (3)
Model\SentencePieceBpeTokenizer.cs (3)
1421if (!SegmentFromEnd(symbols[index].pieceSpan, text, ref textIndex)) 1520return SegmentFromEnd((merge.RightIndex, merge.RightLen), text, ref textIndex) && SegmentFromEnd((merge.LeftIndex, merge.LeftLen), text, ref textIndex);