2 references to FallbackToByteEncoding
Microsoft.ML.Tokenizers (2)
Model\SentencePieceUnigramModel.cs (2)
1135FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount); 1203FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount);