FindSpecialTokenId

3 references to FindSpecialTokenId

Microsoft.ML.Tokenizers (3)

Model\SentencePieceUnigramModel.cs (3)

119CheckSpecialId(addBos, FindSpecialTokenId(ValidateVocab(pieces, unkId), "<s>"), "addBeginningOfSentence"), 120CheckSpecialId(addEos, FindSpecialTokenId(pieces, "</s>"), "addEndOfSentence"), 121FindSpecialTokenId(pieces, "<pad>"), prefixTokens: null, suffixTokens: null)