3 writes to BeginningOfSentenceToken
Microsoft.ML.Tokenizers.Tests (3)
BpeTests.cs (3)
903
BeginningOfSentenceToken
= "<|sos|>",
929
BeginningOfSentenceToken
= options.BeginningOfSentenceToken,
999
bpeOptions.
BeginningOfSentenceToken
= "<｜begin▁of▁sentence｜>";
2 references to BeginningOfSentenceToken
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
196
options.
BeginningOfSentenceToken
,
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
929
BeginningOfSentenceToken = options.
BeginningOfSentenceToken
,