View Single Post
  #1  
Unread 11-30-2017, 01:50 AM
Alex Pepple Alex Pepple is offline
Administrator
 
Join Date: Dec 1999
Location: San Jose, CA
Posts: 4,916
Blog Entries: 140
Default Terrible Hack = Long Downtime

Dear Eratosphereans,

I'd stopped posting news of hacks and recovery for some time now due to an apparent lack of interest in those posts of woe, and the rather esoteric technical explanation of what's going on. But given how badly we were hit this time, and how long a downtime resulted from it, I figure a word or two on it won't hurt.

At some point in my recovery attempt it looked liked like all was lost and I'd have to load the forum software afresh and thereby destroy all the recent member data and posts, and all the years of archived data. I persevered with it, though, for several days and it looks like, in the final analysis, we didn't take too bad a hit in terms of data loss. Still, it's possible that a couple of days or so of member posts might be missing (but it could have been much much worse here too, as is now the case with the Able Muse Press website that's lost a couple of years of data!).

The downtime got extended this time because in the process and steps I took recovering from the hack uncovered some software bugs in the vendor-provided server software, and the ISP/web-host wasn't much help in fixing it, so I had to figure things out myself, but then the server software kept crashing each time I thought I'd recovered and stabilized the system, and each time it crashed, it kept corrupting the data from the member posts, so much so that that data appeared unrecoverable for a long stretch of time--essentially, a perfect storm of hackers, poor support, bad software and software bugs! ... And this went on for days, resulting in the several days of total server failure and downtime. That's about the gist of it--luckily and with lots of debugging and patching work, we're back!

Cheers,
...Alex
Reply With Quote