2 references to FallbackToByteEncoding
Microsoft.ML.Tokenizers (2)
Model\SentencePieceUnigramModel.cs (2)
849FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount); 917FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount);