2 references to AppendTokenWithOffsetAdjusting
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (2)
438
AppendTokenWithOffsetAdjusting
(hit, tokens, offset, addPrefixSpace);
468
AppendTokenWithOffsetAdjusting
(result, tokens, offset, addPrefixSpace);