TryDecodeAsSpecialToken

5 references to TryDecodeAsSpecialToken

Microsoft.ML.Tokenizers (5)

Model\SentencePieceBpeTokenizer.cs (5)

1566TryDecodeAsSpecialToken(this, enumerator.Current, considerSpecialTokens, ref sb); 1585TryDecodeAsSpecialToken(this, enumerator.Current, considerSpecialTokens, ref sb); 1594TryDecodeAsSpecialToken(this, enumerator.Current, considerSpecialTokens, ref sb); 1609TryDecodeAsSpecialToken(this, enumerator.Current, considerSpecialTokens, ref sb); 1645TryDecodeAsSpecialToken(this, enumerator.Current, considerSpecialTokens, ref sb);