today at 2am approx, the linux-noob.com server had a kernel panic. I discovered this when I tried to browse the site this morning, I couldn't access it at all via http or ssh. Ping worked however. So I went to the server and hooked up a CRT and I could see the kernel panic details (some big page fault...)

after a restart dmesg was full of errors (missing inodes and such)

after the restart mysqld was hanging at 100% and linux-noob.com (this site) was non-functional (timed out)


I attempted to backup the current database before attempting to restore it from the last known good backup (from last sunday, as I have a cron job to backup everything once a week)

and the backup attempt failed



because of this, I decided to forgo 3 days data (mon/tue/wed/) and attempted a restore from the last known good backup, that worked just fine and the site is back up and running again.

because of this failure I will now put in place two things,

daily mysqld backup and I will replace this server with a newer one (shortly).

cheers
anyweb

You could have tried using the inbuilt repair commands of MySQL:



Unfortunately now that we have gone back the way and reopened it, then we will end up in a different sort of loss .. hehe So maybe now what we can do is compare your old dump against a new dump and then manually reinsert the differences (question though is what sort of issues would that cause!)

Oh well .. lets just get on with it, I'm certain the loss is minor.

well as the new dump is replaced with the last backup (from sunday) their is effectively no new dump, i did attempt to backup (today) before restoring from sunday but that failed as i documented above.

so, it was three days data of posts from linux-noob.com only, the other databases were not affected.

cheers

anyweb