EncodeWithCache

2 references to EncodeWithCache

Microsoft.ML.Tokenizers (2)

Model\BPETokenizer.cs (2)

338EncodeWithCache(textSpanToEncode.Slice(split.Offset, split.Length), tokens, split.Offset, ref priorityQueue); 343EncodeWithCache(textSpanToEncode, tokens, 0, ref priorityQueue);