3 references to Segment
Microsoft.ML.Tokenizers (3)
Model\SentencePieceTokenizer.cs (3)
360Segment(symbols[index].pieceSpan, text); 440Segment((merge.LeftIndex, merge.LeftLen), text); 441Segment((merge.RightIndex, merge.RightLen), text);