Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5 DOWNTIME !

hi guys


last night (Friday April 14th 2006) the site experienced a meltdown (the filesystem had a major hiccup)



Quote:because of the file system problems, and because i used a backup of the site from April 9th 2006, some members posts/details are lost from the last week or so, apologies
i first noticed it when i checked on the site this morning, and instead of the normal website, there was mysql errors, which pointed to 'read_only' .... 

Quote:mySQL query error: SELECT id, member_id, running_time, location FROM ibf_sessions WHERE id='331641a6b33c5cec3bf0d4bc6f9bbaa6' AND browser='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7' AND ip_address='' 

SQL error: Incorrect key file for table 'ibf_sessions'; try to repair it

SQL error code:

Date: Saturday 15th of April 2006 09:20:30 AM



i ssh'd in and sure enough there were strange things happening, if i tried to create a file (touch test.txt) it would generate an error, read_only filesystem.


touch: cannot touch `test.txt': Read-only file system



This was bad, so i immediately ftp'd over the latest backup to the backup server, and rebooted the box, after hundreds of 'inode errrors' the system finally cam back up, but after some quick testing i could see it was borked, mysql wasnt working and files were missing left right and center


luckily i had a backup web server ready (well nearly ready), installed with fcr4. So I compiled apache/mysql and php and then restored the last backup from April 9th


so, now the site is running on the backup server, please test it and tell me what is/is not working


it'll take a few days to iron out the leaks, and i'll get the other webserver reinstalled.


and yes, it's probably the last time i'll use LVM as a file system !! when something goes wrong, the whole system takes a nose dive




sorry for the downtime !




Hats off to anyweb for getting LN back up so quickly :)

thanks hybrid


just doing my job,


i really want to be availble to all 24/7


so when there's a problem, i'll fix it






Forum Jump:

Users browsing this thread: 1 Guest(s)