_p50kBaseRegex

1 write to _p50kBaseRegex

Microsoft.ML.Tokenizers (1)

Model\TiktokenTokenizer.cs (1)

1191internal static Regex P50kBaseRegex() => _p50kBaseRegex ??= new Regex(P50kBaseRegexPattern, RegexOptions.Compiled, TimeSpan.FromMilliseconds(PreTokenizer.DefaultTimeOutInMilliseconds));