EncodeInternal

2 references to EncodeInternal

Microsoft.ML.Tokenizers (2)

Model\CodeGenTokenizer.cs (2)

391EncodeInternal(null, textSpanToEncode.Slice(split.Offset, split.Length), tokens, addPrefixSpace, split.Offset, agenda); 396EncodeInternal(addPrefixSpace ? null : (normalizedText ?? text), textSpanToEncode, tokens, addPrefixSpace, 0, agenda);