2 references to EncodeToTokens
Microsoft.ML.Tokenizers.Tests (2)
LlamaTests.cs (2)
516encoding = sentencePieceBpe.EncodeToTokens( 681encodedTokens = tokenizer.EncodeToTokens(s, out normalizedText, addBeginningOfSentence: false, addEndOfSentence: false);