EncodeToTokens

5 references to EncodeToTokens

Microsoft.ML.Tokenizers.Tests (5)

LlamaTests.cs (2)

516encoding = sentencePieceBpe.EncodeToTokens( 681encodedTokens = tokenizer.EncodeToTokens(s, out normalizedText, addBeginningOfSentence: false, addEndOfSentence: false);

UnigramTests.cs (3)

337result = _unigramTokenizer.EncodeToTokens(inputText, out normalized, addBeginningOfSentence: true, addEndOfSentence: false); 345result = _unigramTokenizer.EncodeToTokens(inputText, out normalized, addBeginningOfSentence: true, addEndOfSentence: true); 354result = _unigramTokenizerWithSpecialTokens.EncodeToTokens(newString, out normalized, addBeginningOfSentence: false, addEndOfSentence: false);