TryDecodeAsSpecialToken

5 references to TryDecodeAsSpecialToken

Microsoft.ML.Tokenizers (5)

Model\SentencePieceBaseModel.cs (5)

219TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb); 234else if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token)) 239else if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token)) 257TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb); 287if (!TryDecodeAsSpecialToken(this, current, considerSpecialTokens, ref sb) && TryMapIdToToken(current, out string? token))