2 references to FallbackToByteEncoding
Microsoft.ML.Tokenizers (2)
Model\SentencePieceUnigramModel.cs (2)
899FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount); 967FallbackToByteEncoding(ids, normalizationSpan, unknownTokensTracking!, unknownTokensCount);