1 write to _tokenizer
Microsoft.Extensions.DataIngestion (1)
Chunkers\DocumentTokenChunker.cs (1)
37_tokenizer = options.Tokenizer;
4 references to _tokenizer
Microsoft.Extensions.DataIngestion (4)
Chunkers\DocumentTokenChunker.cs (4)
58int contentToProcessTokenCount = _tokenizer.CountTokens(elementContent!, considerNormalization: false); 62int index = _tokenizer.GetIndexByTokenCount( 79contentToProcessTokenCount = _tokenizer.CountTokens(contentToProcess.Span, considerNormalization: false); 103int index = _tokenizer.GetIndexByTokenCountFromEnd(