Update 11:36am
I just spoke to Rackspace again to see how the situation is coming and I was given some bad news. They said they will NOT flip the switch again to even try to route traffic to us until midnight tonight. This means all sites will be down until at least midnight. They said incoming traffic was coming in at the rate of 4GB per second. To put that into perspective our normal incoming traffic for a single day is less then 4GB with outgoing traffic of about 100GB.
As a precaution against even further extended downtime what we are going to be doing now is to begin preparations to move sites individually to new temporary servers so that we can isolate the attack further. If after midnight the attack continues we will start the migration. You are probably asking yourself why can’t we do this now? It would have virtually no effect if done now because DNS settings are by default stored for about 24 hours. The planning stages now involve telling the internet to check for DNS changes more often so that the move can be done tonight as quickly as possible if needed. In addition Rackspace has cautioned against moving it now due to the possible start of another attack.
Sorry if my description is to technical just trying to keep everyone in the loop about whats going on.
Update 10:09am
Please note that all sites are still down and there is very little that we can do at this point except wait. Rackspace has indicated that our site traffic has been null-routed, meaning they are sending it nowhere and they will NOT turn it back on until the attack subsides since the traffic from the attack was so great that it was actually slowing down their entire network.
I have attempted to speak to Rackspace about moving the sites elsewhere temporarily or to a different server in attempt to get everyone back online quickly and they have told me that we cannot make any changes because then the attackers will just attack the new location and it will start again. So we are between a rock and a hard place at this moment. I’m sure everyone else is just as anxious as I am and wanting to get this back online as quickly as possible but right now all we can do is twiddle our thumbs and wait.
I would like to personally apologize to all of our merchants, I am very sorry about this and I really wish I could do something at this point. I’m not sure who is attacking us or why. In 10 years of being an ecommerce shopping cart provider this is a first.
Please bear with us and we will get everyone back online as quickly as we possibly can and I’ll post any further updates here.
Update 9:41am
I am speaking with our hosting provider currently and what has happened is that they have purposefully taken the sites servers and network ips offline because we have received a deluge of traffic that was threatening bringing the Rackspace network down for many other customers as well. They are at this point directing all site traffic to nowhere in an attempt to get the hackers to realize that its going no where and to stop what they are doing. They indicated that normally attack traffic goes down at this point but it has only increased. They are waiting for the traffic to decrease before they essentially turn it back on. I am very sorry, we are doing our best at this point to try and get this resolved as quickly as possible and to keep everyone updated.
Update 9:29am
The admin interface is currently working however sites are not, we are working with Rackspace to get this resolved as quickly as possible and yes this is related to the denial of service attacks.
Update 9:05am
All sites are now running again, we are investigating.
Update 8:35am
Coincidentally, just a few minutes after writing the above message all services went down. We are currently working with Rackspace to determine what has happened and so far it appears that the firewall is not working which is causing all sites to be unavailable. Ie for security purposes traffic cannot go directly to the sites it must go through the firewall. As soon as I have further information I will post it here.
Original Message (now completely invalid but Ill keep it below)
I’m happy to report that all services today are back to normal response time and attack traffic has been steadily subsiding. Response time today is back to normal levels with an average load time today of 0.301 sec and no large spikes. Special rules are still in place to block certain types of connections and we are continuing to monitor the situation and stay on top of any new developments.