TryDecodeAsSpecialToken

5 references to TryDecodeAsSpecialToken

Microsoft.ML.Tokenizers (5)

Model\SentencePieceBaseModel.cs (5)

236TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb); 251else if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token)) 256else if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token)) 274TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb); 304if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token))