2 references to GetToken
Microsoft.ML.Tokenizers (2)
Model\CodeGenTokenizer.cs (2)
1665result.Add(GetToken(value.Id, value.Token, symbols[index].pieceSpan.Index, symbols[index].pieceSpan.Length, originalText, mapping)); 1669result.Add(GetToken(UnknownTokenId.Value, UnknownToken!, symbols[index].pieceSpan.Index, symbols[index].pieceSpan.Length, originalText, mapping));