1 reference to P50kBaseRegexPattern
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1199
[GeneratedRegex(
P50kBaseRegexPattern
, RegexOptions.None, PreTokenizer.DefaultTimeOutInMilliseconds)]