1 reference to P50kBaseRegexPattern
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1181
[GeneratedRegex(
P50kBaseRegexPattern
, RegexOptions.None, PreTokenizer.DefaultTimeOutInMilliseconds)]