2 writes to EndOfSentenceToken
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
948
EndOfSentenceToken
= "<|eos|>"
974
EndOfSentenceToken
= options.EndOfSentenceToken
2 references to EndOfSentenceToken
Microsoft.ML.Tokenizers (1)
Model\BpeTokenizer.cs (1)
197
options.
EndOfSentenceToken
);
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
974
EndOfSentenceToken = options.
EndOfSentenceToken