This site is completely integrated into the Simple Machines forum software. The Wiki account system, the EDC, the issue tracker: all of these use the SMF developer API to work with accounts. I don't really feel like recoding them, and I haven't heard any volunteers.
Moreover, "It's ugly" is never reason not to use anything. Ugly can be fixed. We have complete control over the templates, including the private message box. Your problem with the private message system is an option which you are capable of enabling yourself—it just isn't the default. You can enable it by selecting the As a conversation view instead of All at once. As a special note, if you don't save a copy of your message in your outbox (also not the default), it won't display in the conversation. You can enable doing so by default from your PM settings, too.
I'm guessing MyBB doesn't support that, since you've never heard of it before. User preferences aren't really an uncommon feature of SMF, as opposed to in hacked-together, 20KB pieces of forum software I could write myself in a couple of hours.
That said, if you have any legitimate reasons not to use SMF, go ahead and post them.