Host Europe Group's 123-Reg Screws Up Royally, Accidentally Deletes Scores of VPS

DomainBop

Dormant VPSB Pathogen
The UK's largest host (according to their press releases), 123-reg, which hosts 1.7 million websites and has 800,000 customers, made a boo-boo and accidentally deleted 67 nodes containing hundreds of VPS. The company claims it wasn't that bad because "The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted".

"As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database.


"An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers."
Mirror article: http://www.mirror.co.uk/tech/customers-furious-web-hosting-company-7780626


ZDNet: http://www.zdnet.com/article/web-host-123reg-wipes-hundreds-of-websites-off-the-face-of-the-earth/


 


VPS Outage



Created: 16 April 2016, 09:43 Last Updated: 18 April 2016, 15:42


Some customers on our VPS platform may currently experience connectivity issues. Our engineers are aware of this issue and are currently in the process of resolving this.
We apologise for any inconvenience this may cause.
We will provide a further update as soon as this has been resolved. 
UPDATE 1: 
Sys Admins are still investigating and we have all hands on deck to get this fixed.
UPDATE 2: 
This morning, our teams were made aware of issues affecting the performance of our VPS product. Our teams are continuing to investigate which has led to some customers experiencing denigrated service levels. 
Our tech teams are working hard to rectify for all customers and we thank you for your patience. 
123-reg would like to apologise to customers affected and will continue to provide further updates.
UPDATE 3:
We are still working on finding appropriate solutions for the affected VPS customers. We have allocated extra resources to expedite this. Thank you so much for the patience.
UPDATE 4:
Our teams are continuing to work to restore affected VPS services back to normal throughout the evening and night, using both internal and external experts. 
If you are currently offline and would like to restore from your own backup to save time we can set you up a new VPS image. Please let our support teams know. 
UPDATE 5:
An email will be sent shortly to each customer effected with more information on each case. We will continue to use the status page for main updates and emails to individual customers.
UPDATE 6: 
Customers that have a local backup of their VPS are advised to rebuild their servers following the instructions in this support article: https://www.123-reg.co.uk/support/answers/How-to-rebuild-your-123-reg-VPS-5130/ 
A more comprehensive update in regards to the undergoing work on bringing the servers back online will shortly follow. 
UPDATE 7:
We are currently working on restoring your VPS packages using data recovery tools; in other words, we are attempting to recover your packages bit by bit, while checking data integrity at the same time. 
The process is quite time consuming and the undergoing work is being done in parallel on multiple servers, in several locations. 
Due to the nature of this process, we are unable to prioritise the recovery of specific hosts, but we are focused on restoring bulks of data at the moment. Progress is visible, and we have already managed restoring a certain number of hosts. However, we are unable to estimate when will the recovery be finished on all of the affected packages. 
We can assure you that our teams are doing everything possible in order to get back on track. 
Please do note that response times on our communication channels might be a bit higher than usual, at the moment. 
We would like to thank you for your patience on the matter. 
UPDATE 8:
As the data recovery work continues for those remaining VPS we ask that you contact our service teams through the email address : [email protected]
The team reviewing these emails are able to see the status of the work so far and update you where possible. 
Please note that we are actively updating all outstanding customers on a case by case basis as we get information from the recovery team. This update will be via email. We would ask that you wait for the email update. If you have questions about the recovery or need other information please do email the team.


https://www.123-reg.co.uk/support/system-status/
I love update 2: "This morning, our teams were made aware of issues affecting the performance of our VPS product." <--deleting your customers' VPS's is a performance issue? LOLLLLLLLLLLLLLL


123-reg is owned by Host Europe Group...in their own words:

HEG is now Europe’s largest privately owned hosting provider, largest domain registrar in the UK and Germany’s largest provider of Managed Hosting; Number 2 in Managed Hosting in Europe. We currently own 7 data centre facilities in Europe and USA, supported by over 1,000 employees. HEG has more than 1.7 million customers with over 7 million domain names under management. This makes HEG Europe’s second largest hosting provider.
TL;DR major screwups aren't limited to little 1-man shows run out of mommy's basement
 
Last edited by a moderator:
Top