At around 20:45 (EST) on Oct 1, we began experiencing another similar SMF Database error which took down the entirety of the enigma-dev website. We informed Gary, and he came on at around 1:45 and resolved it immediately.
The issue was that we overflowed our httpd error logs, exceeded our disk quota, and were unable to assign new database pids. The fix was simply to delete the logs, freeing up 30g of space.
This leaves us with 2 pieces of good news, and 1 piece of bad news.
The good news is that it's very easy to fix, and should it happen again, we can fix it much quicker without needing to wait for gary.
The bad news is that, since we didn't look at the logs, we're not really sure what caused them to overflow in the first place, so it could happen again in the semi-near future (probably a couple weeks from now)
The other good news is that if and when it happens again, we'll take a peek in the logs and hopefully be able to fix it for good.