News  -  Press Releases  -  Chat Now  -  Forums  -  IRC Network  -  IRC Help  -  About ETG  -  Contact Us
   Currently On ETG  

 
   
General News  
ETG News - Recent
ETG News - Archives
Press Releases
Forums
About EnterTheGame  
Company Info
Employees
Contacts
EnterTheGame Realms  
Doom III
CounterStrike
Quake IV
Full Directory
IRC Network Information  
News
News Archive
Organizations
How To Join / Chat
ETG Rules & Policies
ETG`s Purpose / Mission
ETG`s Server List
Partners
Network Staff
Network Helpers
How To Join The Staff
How To Link a Server
IRC Help and Guides  
New User Guide
IRC Etiquette
FAQs & Resources
ChanServ Help
VHOST Info
Hostmask Help
mIRC Setup Info
Security
Chat With Admin
Scripting
Hurricane Sandy & Channel Services Outage November 8th, 2012
An Important Message To All ETG Users Regarding The Ongoing Outage Since Hurricane Sandy From David Sherman (WHiZ):

I wanted to take a moment to issue a personal apology to all of our thousands of loyal users and friends for the recent outage. I`m Sorry, and I take full responsibility for it, and will continue to work to restore all of our services ASAP.

During Hurricane Sandy, our most critical server that operates and provides support to all of ETG`s various Channel Services literally got *FRIED*.

All of our databases/software and all of your channels are safe, and securely backed up and we have lost no data what-so-ever because of good planning and infrastructure.

Normally in the case of a hardware failure, I`ve always run out to the store, bought the parts, and replaced them and had things back up in a matter of hours. In cases of outages of fiber cuts or power cuts, I`ve likewise gone out of my way even going out with tornadoes in the area, tornado warnings, severe storms, ice storms etc to get to our datacenter and get power patched over to other sources or whatever else back up asap.

We`ve had a disaster recovery plan and system in place for many many years, that would bring our services back up in a backup mode in case of any catastrophic outage until we got the main systems repaired. But as much as I`ve searched I`ve been unable to locate it. As bad as that sounds, it`s the truth, and I`m sorry for that. I`ve searched a number of times including this morning hoping to find it so i could restore service quickly/easily. Sadly all to no avail, clearly I made a mistake and have misplaced it somehow. When you build something or put something in place and then never have a need to use it for years on end, i guess it can be your undoing.

Also, I generally don`t talk about myself to the public. But my health hasn`t been well the past couple years, and is in fairly bad shape right now. The reality of the matter is it has left me unable to provide the same level of effort and concentration on this outage as normal. I have been working on it pretty much non-stop daily, but the progress has been slow going for a variety of reasons. The largest of those reasons, I fear has been because of the state of my health.

We`ve been consecutively down more now, than I can ever remember. For that, I am beyond sorry. I`m also sorry for letting you all down with us not meeting our expected level of high up-time. It actually breaks my heart to know that we`ve failed you in that pledge right now. But please be assured it *IS* being worked on to the best of my ability right now.

One of the other reasons things have been slow going, was a good suggestion by one staff member to make our services more virtual/cloud based to help with such situations in the future. Sadly that has been a little easier said than done. I`ve made some good progress, but it`s been very slow going.

My 3 options the past week have been 1) remake the disaster system (probably would take about a day), build a new system and get things up (probably take about a day), or finish moving things to a virtual cloud based solution(How long could that take???).

To this point in time, besides a lot of searching for the backup system Ive been working mainly on the hardware\low level aspects of option #3, and I suppose rather naively assumed it wouldn`t take as long as it has. At first the setup showed great promise and it looked like a perfect solution. Through-out the process I kept thinking I was getting close and it would work, and each time the hardware\low level items have proven to have this or that issue and have not be co-operative despite everything seeming good. Then i had to research or figure out work-arounds. At any point I could have switched to a different option, but I naively kept thinking it would only take a little longer, aka. "I just have to fix this one issue and it will be ready." etc. Sadly, after each issue got solved, i only found another one that also existed.

I`m sure we`ve all been in those situations. But I clearly errored in my judgement, and I want to apologize for that. I should have cut our losses and just put the effort into getting one of the other solutions up and operational for while i kept working on the final solution.

So, alas, we come to the important question of "What now?" Well, if you could please give me a little more time to finish what i`m currently working on to see if it works or if there are still other issues, I`d appreciate it. If I`m unable to coax it into working by this evening, I promise I will stop working on that and instead start working on a different solution rather than perpetuating the outage any longer.

Again, I`m sorry for this. The outage, while caused by the hurricane, has gone on longer than it should have due to my health issues and my errors in judgement and nobody elses. I`ve been working every day to the best of my ability to restore the service and it has been very frustrating for me and my fellow admins too, not just to you.

I`d like to thank you for your understanding and continued patronage. All of the staff here is truly appreciative of your patience with the matter. We pledge to have service back up sometime tomorrow at the latest.

Sincerely,
David Sherman (WHiZ)
Founder of Enter The Game
5419 characters
Other Site News
Hurricane Sandy & Channel Se...
ETG Celebrates 13 Years!
DNS Outage
Maint on PA Server
IRC is getting an UPGRADE!
MD Server Maint
New Feature for Approved Egg...
Eggdrop Issues
ETG Celebrates 10 Years!!!
Reserved Nicknames
New Security System Enabled
Happy 9th Birthday ETG!
Major Network Maint TODAY!
UT3 Developer Chat TODAY!!!
9/11 Remembered
In The Forums
Site design and contents copyright (c) EnterTheGame & Excelsior Media Studios, Inc. 1999-2017
No portion of this site or any content on it can be copied or otherwise used/reproduced elsewhere without explicit permission by Excelsior Media Studios, Inc.
Site Coded/Constructed/Designed by Dave Sherman @ Excelsior Media Studios, inc.
Original Site Concept & Design by Callidus @ Digital Impurity - www.digitalimpurity.com