1 write to EndOfSentenceId
Microsoft.ML.Tokenizers (1)
Model\BPETokenizer.cs (1)
339EndOfSentenceId = aId;
2 references to EndOfSentenceId
Microsoft.ML.Tokenizers (2)
Model\BPETokenizer.cs (2)
476tokens.Add(new EncodedToken(EndOfSentenceId, EndOfSentenceToken, new Range(charsConsumed, charsConsumed))); 543ids.Add(EndOfSentenceId);