3 writes to BeginningOfSentenceToken
Microsoft.ML.Tokenizers.Tests (3)
BpeTests.cs (3)
947
BeginningOfSentenceToken
= "<|sos|>",
973
BeginningOfSentenceToken
= options.BeginningOfSentenceToken,
1043
bpeOptions.
BeginningOfSentenceToken
= "<｜begin▁of▁sentence｜>";
2 references to BeginningOfSentenceToken
Microsoft.ML.Tokenizers (1)
Model\BpeTokenizer.cs (1)
196
options.
BeginningOfSentenceToken
,
Microsoft.ML.Tokenizers.Tests (1)
BpeTests.cs (1)
973
BeginningOfSentenceToken = options.
BeginningOfSentenceToken
,