2 references to AppendTokenWithOffsetAdjusting
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (2)
441
AppendTokenWithOffsetAdjusting
(hit, tokens, offset, addPrefixSpace);
471
AppendTokenWithOffsetAdjusting
(result, tokens, offset, addPrefixSpace);