Big Server Crash, Big Downtime

Alex Pepple · #1 01-16-2019, 04:39 AM

Dear Eratosphereans,

I'm sure most of you have been concerned about what turned out to be one of the longest downtime for the Sphere, and wondered if we'll be up again? Well, we're back!

And maybe you’re all a bit fatigued with this report … and truth be told, maybe I’m there too, and not just the report but all the work involved in all that salvaging and restoration! Still, I've finally got a bit of sleep after quite a few all-nighters ... and here comes another report …

And this time, it's not the hackers directly but I can say it's them indirectly because one of the additional security applications I installed to keep the hackers out misbehaved badly and has crashed the server, specifically, the operating system that controls just about every other program. For the crashed security software, I directly blame the maker of the server software who also supplied that security tool that turns out to a poorly designed program! Essentially, it killed the server to the point where it just went down and won't even boot! But back to the hackers--maybe, it's still them also, and they’d placed a time bomb somewhere that kicked in and caused the huge crash!

Anyhow, that's were things were, and I'd worked feverishly for days on end to get things back up. The task involved buying/leasing a brand new server; getting it loaded and configured as close to what I remember the old one was set to; and then try to move over all the data from the dead system to the new server, and reconfigure everything well enough to work again as close to how it worked before as possible. It if sounds daunting, well ... it was! And part of the reason this took way too long is because the server's hosting company were just not too helpful or rather, just not knowledgeable enough and kept setting the new server wrong (I was at their mercy because there was no option they'd provided for me to do it myself). Anyhow, each redo of the setup they made took almost all day! Eventually, I lucked out and stumbled on someone there smart enough to set thing exactly how I'd specified. Going back to the incompetence part, I truly believe that there would not have been a need to abandon the old server and set up a new one and try data recovery if only their technical staff knew enough how to fix the boot system on the old server (I don't have access to that after a crash and I can only rely on them!). They claim that they tried and got some error message they couldn't quite figure out or knew how to resolve, and then they offered to reinstall the operating system afresh, but in the process they'd wipe out all the user data (i.e., Eratosphere would become a blank slate!). That was, of course, unacceptable. And I'm quite certain a knowledgeable technical staff would have readily fixed the boot problem and avoided the days of delays that I then needed to set up a new server and attempt to recover the old data over to it. Well, not to bore you with details, but all has ended well, at last!

Still, I'm sure that there are some settings and configurations that don't quite match what we have before, and that will manifest itself in a few functions, a few things falling apart when you try to use them. So, if you note any problems, just report it in this thread, or else PM me.

Moreover, just to make sure we never go through this type of protracted downtime again, either because of hackers or flaky server and third-party software, I'll be purchasing subscription space from an external cloud storage company where I'll be daily uploading a complete image of the server--yes, more expenses! Anyhow, with the external/remote image backup in place, all I'll have to do then, should things henceforth break badly locally, is recreate everything from the external server image--which should only take hours instead of days.

Finally, I'm happy to report that there was no data loss this time (or at least, there shouldn't be!) as I managed to recover everything this time--from the moment of the crash all the way to the very beginning.

That's most of it. Thank you all for your patience during the downtime, and for your continued support!

Cheers,
...Alex

John Isbell · #2 01-16-2019, 04:52 AM

Can someone post the donation link again? It didn't pop up on my phone just now. :-)

Thanks,again, Alex!

Cheers,
John

Alex Pepple · #3 01-16-2019, 06:27 AM

Hey, John--Good to see you here ... after the restoration.

Donation link is an easy one: click the donate link near top left of most Erato pages or simply click here.

Thanks again!
...Alex

John Isbell · #4 01-16-2019, 07:54 AM

Thanks Alex! Donation made. :-)

John

Alex Pepple · #5 01-16-2019, 08:04 AM

Thanks so much, John, for the support ... much appreciated!

Cheers,
...Alex

Allen Tice · #6 01-16-2019, 09:32 AM

If you don’t want to glow in the dark as a “muse”, just toss in $350 or something like that, or GBP 200. I know that reads like I’m showing off, but I ain’t, dang it! I got a year-end bonus, and WTF, all you ugly bums out there and svelte damosels are important to me—well, most of you, most of the time. Even so, during the silent time, I got stuff done that was worth that much to me in the long run. Cough up. I almost shut down my own website because it’s getting expensive. Money isn’t worth what it used to be worth. If Eratosphere goes, whom can you berate?