ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[mnsalem]

Finally!y VPS is back online! 3 hours after my ticket was out into restoration category! The backup I think is around June 10th to June 14th .. On BUF19


But hey, at least they did have backups after all!


Good luck to everyone else getting your data back!

 

[mnsalem]

the latest email received from CVPS - Report #10

First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario. We are absolutely dedicated to ensuring the happiness and recovery of every single impacted customer.

Now that we’ve begun to get ahead of the support load I’m able to share more specific details on our current status. At this time all customer services should be online and available. A small subset of customers may still be offline as their containers will require further investigation and attention. A few nodes have been condensed/transferred/migrated so please do not be alerted if your node ID has changed. Direct access to SolusVM remains inactive as we wait for their internal and external security audits to be completed (as discussed here: http://www.lowendtalk.com/discussion/11327/solusvm-audit-update#latest). During the interim you are able to control your containers through our billing/support system and may request OS reloads via ticket.

Initially the mitigation strategy called for all impacted nodes to be inspected for logs, reloaded, and then images from our backup database installed. This process proved far too timely and dramatically slowed down the recovery effort which is also partially to blame for our poor communication during the first 48 hours post-compromise. We altered the plan two days ago, deciding to instead install all impacted customer service from scratch; this revised strategy has allowed us to rapidly return to full online status and reduce pressure as we work diligently to recover data for all customers who require it. ChicagoVPS has two separate backup facilities, a free public facing system called Central Backup and a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted. For customers on those nodes, if you ran a Central Backup your data is absolutely safe and you may request a restoration via ticket. Customers who were not on the nodes with corrupted secondary backups can request a restoration regardless of whether they utilized the free Central Backup feature. The restoration process currently has a 6-12 hour lead time once you’ve requested it.

ChicagoVPS will be analyzing this event closely so we may implement refined plans to both protect against repeat issues and to ensure our communication and reaction strategies are improved. While SolusVM has released multiple updates in the past few days to fix vulnerabilities which allowed this event, and others, to occur we take full responsibility for our outage. We simply should have reacted more quickly, more effectively and provided better communication. We’ve already learned from this past week, and we will continue to learn.

As we push towards final resolution on all fronts we will continue to update our customers via email, Facebook (https://www.facebook.com/chicagovps) and the “Low End Talk” message board (http://www.lowendtalk.com/discussion/11304/chicagovps-update#latest).

Thank you again for your patience, loyalty and understand. We appreciate you.

Regards,

Chris Fabozzi
Director of Operations
ChicagoVPS

 

[drmike]

Fabozzi never fails to amaze me with his abuse of the English language.

First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario.

 

He calls this SITUATION a scenario? Nothing imagined about it.  Nothing planned.  It is pure post-apocalypse mopping of the floors.  Scenario would be appropriate if he had contingency planning and policies in place which foresaw and had resolution for such an outcome or scenario to happen in the future potentially.








 

​







sce·nar·io
 noun \sə-ˈner-ē-ˌō, US also and especially British -ˈnär-\

plural sce·nar·i·os



 









Definition of SCENARIO

1

a : an outline or synopsis of a play; especially : a plot outline used by actors of the commedia dell'arte
 

b : the libretto of an opera


2

a : screenplay
 

b : shooting script


3

: a sequence of events especially when imagined; especially :an account or synopsis of a possible course of action or events <his scenario for a settlement envisages…reunification — Selig Harrison>



 See scenario defined for English-language learners »



See scenario defined for kids »




Examples of SCENARIO

1. A possible scenario would be that we move to the city.
2. The most likely scenario is that he goes back to school in the fall.
3. The best-case scenario would be for us to finish the work by tomorrow.
4. In the worst-case scenario, we would have to start the project all over again.

Origin of SCENARIO
Italian, from Latin scaenarium place for erecting stages, from scaena stage
First Known Use: 1875





Related to SCENARIO


Synonyms

screenplay, script



 

Related Words

shooting script; story, text

 

a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted.

Small group of nodes?  It's roughly 25% that weren't running backups plus whatever others that had failed/wrong/broke backups.   Small is north of 25%?!?!?!  Pretty lousy numbers from the son of an accountant.

I HATE HATE HATE providers that smokescreen things and put fluff all over the place.  Chris wasn't busy doing the admin work.  He should have been replying to tickets and doing the press / update / news circuit (Facebook, Twitter, LET, etc.)

Maybe the third hack-go-round CVPS will handle it properly ?

 

[MannDude]

Ok, so this is the original thread and IPB staff has restored it. Due to some errors going to have to split the threads up I think. There may be duplicate content in this thread that also exists in this thread here: http://vpsboard.com/topic/758-chicagovps-cvps-hacked-new-solusvm-exploit/

That thread will be locked until I clean it up.