ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[drmike]

but both were on ChicagoVPS

 

There weren't too many hosts smacked by this.  RamNode was the highest profile provider around here and while it took time, I think they rescued everything.

There were something like 1000+ people who had more than one VPS with CVPS.   So, that practice is roughly 10% of their total VPSes deployed.

There are companies who aren't based on SolusVM.  There is an ongoing thread about that on here.   Backupsy comes to mind and BlueVM and BuyVM (I think).

You can cheaply create failover capability with 3 VPSes (different providers) and Rage4 DNS (free depending on your use).  For database security do cron jobbed mysqldumps and scp those off to a remote backup storage like Backupsy.  Ditto for source files, rsync them over to Backupsy.  Synchronizing the database from the failed over to node is another complex story, but minimal impacted users in that scenario.

 

[zulualpha]

@Zero, the recent email explains it all and should have all the fresh installs up within the next few hours. If you want us to try and restore from our backups, you need to open a ticket.

Chris, can we get an update, since 10 hours have passed & it looks like fewer nodes are up now than before. Are you still doing fresh installs?

 

[MannDude]

Chris, can we get an update, since 10 hours have passed & it looks like fewer nodes are up now than before. Are you still doing fresh installs?

Looks like 15 nodes are down right now: http://stats.pingdom.com/jzrszp4wfu79

Though I don't think their Pingdom monitors all of them. The leak revealed they had 109 nodes, which is strange considering the hack back in November revealed about 45-50 nodes (I believe). That was after being in business for many years. So after that hack, their business doubled in less than 9 months? That doesn't make sense. I expect after all this is resolved, that they'll have 250 nodes in 3 months.

 

[MannDude]

This is interesting: https://twitter.com/FrantechCA/status/347533537014075392

@stormandsong Board arrived this morning. ChicagoVPS got hacked so the DC has been busy dealing with that all. They told us 'a few hours'.

CC must be busy with getting LET working halfway properly, and dealing with CVPS. Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.

 

[SeriesN]

I am surprised no one here used Chris's formula and offered "refugee" coupons. 

Peer respect! This is another thing that sets this forum apart.

 

[Aldryic C'boas]

Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.

I'm getting pretty close to the point of becoming vocal about it, aye.   

 

[Mun]

I'm sorry but people are just idiots.

Mun

 

[leeboof]

I really hope they update later tonight or early tomorrow morning as to what the deal is. It went from servers should be up in 24 hours to we had an issue restoring so everything will be fresh and running today to no update.

At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.

 

[MannDude]

Has anyone who had a server with dataloss been brought back online from a CVPS restored/maintained backup yet? I'm curious if there are customers who earlier posted issues that are now resolved.

 

[srichter]

At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.

The problem being "Well we probably already lost a ton of customers so let's just take a fucking break for a bit."

 

[drmike]

Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.

 

Here, I'll say it, hasn't BuyVM had parts on site at Colocrossing since before end of the business day?  By my clock, 8 hours or so right?

 

[drmike]

Remote hands:

Remote Hands (Everything not covered above) $125/Hr

$125/hr    x   24 hours = $3000 per 24 hours 

$3000 per 24 hours    x   number of admins allocated = $6k for 2  $9k for 3....

Minus any discount

Average income per VPS can't be very high based on pricing.  Probably $3.50 per VPS.

$3.50 x 9000 VPSes = $31,500 a month.

3-4 days of round clock outsourced to CC hands = negative income month.

 

[Francisco]

Here, I'll say it, hasn't BuyVM had parts on site at Colocrossing since before end of the business day?  By my clock, 8 hours or so right?

Motherboards isn't something we kept spare in NY since we have so few KVM nodes there.

We missed the "nest day window" on amazon by a few hours so we lost a whole day.

Francisco

 

[XFS_Duke]

I really hope they update later tonight or early tomorrow morning as to what the deal is. It went from servers should be up in 24 hours to we had an issue restoring so everything will be fresh and running today to no update.

At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.

They are restoring accounts now. They are setting up new VPS accounts for most people. I know this for a fact, so just take a chill pill. There are a lot of servers that they weren't able to just "restore". If you want yours restored, submit a ticket and they'll get to it. If you want your VPS back up and you have backups yourself, then maybe we can work something out. For now, they're running through each node recreating accounts. Just give them time...

 

[drmike]

Sad that folks not in the know and just customers feel ignored, abandoned, etc.

• 
  
  
  
  
  Cheysser Estrella Valdez Update please
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
  
  
   
  
  
  
  Drew Read Update please...!
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
   
  
  
  
  Joel DeVenney This is CRAZY!!! Two days and counting, going on 3 now and their last email they promised to have everything back online today!!!
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
  
  
   
  
  
  
  Christopher Breen This is ridiculous. No response to tickets, no restoration of any kind to remaining hosts... Seriously amateur hour...
  
  about an hour ago via mobile
  
  
  
  
  
  
  

 

[XFS_Duke]

Sad that folks not in the know and just customers feel ignored, abandoned, etc.

• 
  
  
  
  
  Cheysser Estrella Valdez Update please
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
  
  
   
  
  
  
  Drew Read Update please...!
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
   
  
  
  
  Joel DeVenney This is CRAZY!!! Two days and counting, going on 3 now and their last email they promised to have everything back online today!!!
  
  3 hours ago
  
  
  
  
  
  
  
• 
  
  
  
  
  
  
   
  
  
  
  Christopher Breen This is ridiculous. No response to tickets, no restoration of any kind to remaining hosts... Seriously amateur hour...
  
  about an hour ago via mobile
  
  
  
  
  
  
  

They should be getting emails... Unless their email is being hosted on the VPS... Now, if I were Chris I'd post this on their Facebook as well, if it wasn't already, didn't check...

EDIT: Doesn't look like they did, I posted, maybe someone will read it.

 

[srichter]

I just want to start off by saying thank you to everybody and their patience. I know this situation has been very frustrating and long, and I can assure you we are doing our best. We are still restoring VPS' via support ticket request. While doing this, we have noticed that a small percentage of the backups were corrupted after being restored. Those of you that are wondering why it’s taking so long for us to get to your ticket, I wanted to clear that up. There are a lot of customers impacted and this is a very timely process and the ChicagoVPS team is going through them very carefully.


I want to state again that there will be compensation for all clients affected by the compromise. We would like to thank you again for your patience, and we are getting closer to getting everything back in working order.


Thank you,

The ChicagoVPS Team

 

[upsetcvps]

ramnode, out of curiousity, how many vps went down for you and how long did it take you to get them back up (also if you don't mind saying, how many people worked on it and what sort of recovery procedure did you follow)?  I'd like to compare with cvps and maybe they can even get some tips from you.

 

[infinityhosting]

My vps is still down and I have not had a reply to my ticket. WAA-970698 I opened it 12 hours ago. I just want my vps back up so I can get back in business like everyone else.

NOw 24 hours and no response. I had to get a temporary hosting solution through another vendor. I need my vps up.

 

[kauffjd3]

Still down and no responses to my tickets. 

I started rebuilding on another host.  I completely understand cvps is a victim, but they are the company that are providing a service which failed. 

My friends just came up with all data.  I guess there is still hope. I don't know at this point.