What happened ??

[rolygate]

Explanation time...

The main server with the load balancer crashed due to a hard disk crash that also corrupted the data on the RAID disk, which is the main backup. For some reason not yet explained yet the host's backup was faulty and could not be restored, so data had to be recreated from other sources.

I was ill today and did not do several things that should have been done - and for that I deeply apologize.

Facebook is a good idea, we should have posted there. However there is an unresolved issue with this, we currently have two FB pages and it needs the two admins to sit down and work something out - each page has grown up independently of the other. I think one should close as it must be very confusing to people looking for ECF's page, the one that can be found under ECF is not the 'official' page, which is at e.cigarette.forum - but it is less busy than the other one. Hey hum. Add to that SJ's FB page and it is very confusing at the moment.

Let's hope tomorrow is a better day, today was a bit grim.

 

[Olef]

DNS has been screwed all day. Is the ECF DNS server on the same RAID as the database?

Just interested...

 

[thorn]

The only relief I had was that ping responded to keep the hope alive.

 

[Uma]

Happy Birthday to Augustus!

Just now finally here after trying all morning. My bookmark link still didn't work, so I did a google search for ecf, clicked on a link and voila. I hope others are having better luck than I.

 

[JerryRM]

Thanks for the explanation, Chris and there is no need to apologize if you weren't feeling well today.

 

[Smoking Panda]

thank you ECF Website Manager for yr hard work to restore the ecf .... // bow // bow // ......

restoring PDC from BDC without backups are no fun work .... i been there i knw ....

 

[jlauro]

DNS has been screwed all day. Is the ECF DNS server on the same RAID as the database?

Just interested...

Sort of (don't want to give details away) ... Which made it impractical to point people to a working system as updating root servers could take 24 hours... and the system took significantly longer than expected to get back online.


Steps will be taken so we can handle a similar outage better in the future. It's only been about 3-4 months there was another server to really make splitting the load for DNS and other services practical. No excuse, but it mostly fell into the if it isn't broke... category for the last few months.

Having the DNS split probably wouldn't have helped in the amount of down time, but would of made putting a status page up more practical, and may have also made it feasible to restore and reroute services to one of the remaining servers instead of waiting for rebuild and recover, which could mean lower downtime at the expense of some data loss.

During the entire outage the estimate would of probably always been 2 hours. So having a status page probably wouldn't of helped much.

 

[mlinky]

Thanks for getting back up so quickly, although my ECF withdrawals were pretty wicked.

Feel better soon Roly!

 

[stravaigin]

Explanation time...

The main server with the load balancer crashed due to a hard disk crash that also corrupted the data on the RAID disk, which is the main backup. For some reason not yet explained yet the host's backup was faulty and could not be restored, so data had to be recreated from other sources.

I was ill today and did not do several things that should have been done - and for that I deeply apologize.

Facebook is a good idea, we should have posted there. However there is an unresolved issue with this, we currently have two FB pages and it needs the two admins to sit down and work something out - each page has grown up independently of the other. I think one should close as it must be very confusing to people looking for ECF's page, the one that can be found under ECF is not the 'official' page, which is at e.cigarette.forum - but it is less busy than the other one. Hey hum. Add to that SJ's FB page and it is very confusing at the moment.

Let's hope tomorrow is a better day, today was a bit grim.

Sorry to hear you're not feeling well. Thanks for the explanation. Being without ECF and my Reo friends made me realise how much I rely on this place There are good people here.

 

[Olef]

Sort of (don't want to give details away) ... Which made it impractical to point people to a working system as updating root servers could take 24 hours... and the system took significantly longer than expected to get back online.


Steps will be taken so we can handle a similar outage better in the future. It's only been about 3-4 months there was another server to really make splitting the load for DNS and other services practical. No excuse, but it mostly fell into the if it isn't broke... category for the last few months.

Having the DNS split probably wouldn't have helped in the amount of down time, but would of made putting a status page up more practical, and may have also made it feasible to restore and reroute services to one of the remaining servers instead of waiting for rebuild and recover, which could mean lower downtime at the expense of some data loss.

During the entire outage the estimate would of probably always been 2 hours. So having a status page probably wouldn't of helped much.

Thanks for the info, as I say just interested as IT is my day job. I imagine my day without ECF has been a lot more restful than everyone who keeps the site running so thanks all for your efforts. They are appreciated

 

[JerryRM]

For me anyway, it wasn't that the ECF was down for a day, it was the speculation as to why it was down. From what I hear, there were a lot of rumors flying around.

 

[AngusATAT]

For me anyway, it wasn't that the ECF was down for a day, it was the speculation as to why it was down. From what I hear, there were a lot of rumors flying around.

There always are.

So what was the big one this time? FDA terrorists? Hacked by Phillip Morris?

 

[440BB]

I just figured that ECF wasn't feelin' the love and wanted us to not take him/her for granted. We have all been reminded that ECF is a need, not a want. Now back to the threads so my hands stop shaking...

 

[Pamdane]

I actually had wondered privately if "someone" hadnt found a way to give ECF a virus. Then I thought to check FB, and when no one had responded there, I felt abandoned(still rocking in corner comforting myself now and then).....

I hope your feeling better now

 

[JerryRM]

There always are.

So what was the big one this time? FDA terrorists? Hacked by Phillip Morris?

FDA terrorists ranked high up on the list.

 

[ime5000]

Glad that you're back up

(ECF withdrawal is awful)

no kidding! i checked like every half hour at least!


i thought something bad happened... before e ciggs i was doing real good with snus, till i logged in to by some and that day MD was banned from ordering snus online...

 

[JerryRM]

Personally, I thought that the FDA had raided the ECF headquarters, shut down the forum and had locked up Roly and Angus in j0kers old cell in the ECF dungeon.

Sarge's explanation that a hard drive had crashed, was just a bit too unbelievable to me.

 

[Tendril]

Hey gang,

Thanks for working so hard to get the site back up

You don't have to apologize, but it was nice of The Team. My professional opinion would be that you don't need to disclose any information about your network beyond the internet-facing IP address, though. Saying it was "a hard drive issue" or a "hard drive failure" is more than adequate.

Repeat: Thanks for working so hard to get the site back up!

 

[cracker75]

Thanks for all the hard work Admins!

A little as I contribute, I love my ECF!

 

[Surf Monkey]

Rumors fly, but you can usually tell it's a server issue by the errors it throws. There was only a short time when the server was completely off line.