1 reference to FallbackToByteEncoding
Microsoft.ML.Tokenizers (1)
Model\SentencePieceUnigramModel.cs (1)
452
FallbackToByteEncoding
(normalizedString, tokens, insertionStartPosition);