2 references to EncodeToTokens
Microsoft.ML.Tokenizers.Tests (2)
LlamaTests.cs (2)
516encoding = sentencePieceBpe.EncodeToTokens( 681encodedTokens = tokenizer.EncodeToTokens(s, out normalizedString, addBeginningOfSentence: false, addEndOfSentence: false);