2 writes to EndOfSentenceToken
Microsoft.ML.Tokenizers.Tests (2)
BpeTests.cs (2)
904
EndOfSentenceToken
= "<|eos|>"
930
EndOfSentenceToken
= options.EndOfSentenceToken
2 references to EndOfSentenceToken
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
197
options.
EndOfSentenceToken
);
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
930
EndOfSentenceToken = options.
EndOfSentenceToken