1 reference to Cl100kBaseRegexPattern
Microsoft.ML.Tokenizers (1)
Model\TiktokenTokenizer.cs (1)
1260
[GeneratedRegex(
Cl100kBaseRegexPattern
, RegexOptions.None, PreTokenizer.DefaultTimeOutInMilliseconds)]