1 write to EosPiece
Microsoft.ML.Tokenizers (1)
SentencepieceModel.cs (1)
386
result.
EosPiece
= SentencePieceProtobufReader.ReadString(data, end, ref pos);
3 references to EosPiece
Microsoft.ML.Tokenizers (3)
Model\SentencePieceBaseModel.cs (1)
29
EndOfSentenceToken = modelProto.TrainerSpec.
EosPiece
?? "</s>";
Model\SentencePieceUnigramModel.cs (2)
83
_vocab[modelProto.TrainerSpec.
EosPiece
] = modelProto.TrainerSpec.EosId;
86
_vocabReverse[modelProto.TrainerSpec.EosId] = (modelProto.TrainerSpec.
EosPiece
, 0f, ModelProto.Types.SentencePiece.Types.Type.Control);