_transformedSpace

2 references to _transformedSpace

Microsoft.ML.Tokenizers (2)

Model\CodeGenTokenizer.cs (2)

1304ReadOnlySpan<char> span = firstToken && hasPrefixSpace && s.Length > 0 && s[0] == _transformedSpace ? s.AsSpan(1) : s.AsSpan(); 1466ReadOnlySpan<char> span = firstToken && hasPrefixSpace && s.Length > 0 && s[0] == _transformedSpace ? s.AsSpan(1) : s.AsSpan();