ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[Drar]

I think cvps took today off.

^This...

 

[bellicus]

I think cvps took today off.

Would you put it past them? I'm still waiting for a simple reply :unsure: My 2nd CHI VPS just went dead and it didn't go down at all during the whole mess

 

[upsetcvps]

Yep, plane landed in Brazil by now.

 

[Drar]

You know what guys, I am starting to think that Chicago VPS have lost the backups or don't have any backups of our VPS at all.

I have several tickets opened asking about the about the data restoration but all I get are unrelated canned replies. I will not be surprised if on their next "RFO" they will say something like "Hey sorry but our backups got corrupted etc etc so we will just give you a fresh VPS so you can start from scratch yada yada yada..."

If they don't have backups they might as well come out and tell the truth so that we can have another contingency plan instead of waiting and relying on false hopes. I have couple GBs worth of data and it will be a pain in the a** to upload it using my home internet connection due to slow upload speeds.

Again, this is just me thinking about the "what ifs". Let's just hope that I am wrong about this...

What are your thoughts so far?

EDIT: Just got a confirmation from one of the users here that the data on his VPS has been restored by CVPS.

Will update this post as soon as my VPS has been restored as well.

 

[zulualpha]

I finally got one of my 2 VPS in buffalo back up today - it was a fresh install & the password hadn't been changed. There was no notification to let me know that it was back up. Fortunately I have my own backups, but it's going to be time consuming to upload them all. 

 

[Aldryic C'boas]

Food for thought:  SolusVM has a backup system included.  However, information is stored much the same way as node information is - and if the attacker could wipe the nodes, there was nothing stopping him from wiping even remote backups as well if they were tied into Solus.

Sure, there's always the chance that they used different software, or wrote their own scripts for backups.  But I wouldn't wager on that.

 

[jfreak53]

I just got one of my 10 back up with the backup restored finally! But God it took forever, not to mention, it was still using the old password and they didn't even let me know it was back up!! Even though I have a TICKET open requesting this IP be restored ha ha. None others yet.

 

[upsetcvps]

I seem to be able to reset my root password through wmhcs now.  If your vps is up but you don't have log in credentials, I suggest you do this (and then change your password again once you are logged in using passwd).

 

[jfreak53]

Hahahahaha get this crap, just got it in a new message to a ticket I had open:

At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled. 

We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.

You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc). 

Thank you again for your business and support.

---------------
Matthew
Support Guru

Wait, what?! I thought in the last RFO report they said they had completed ALL node fixing and were now working on restoring? So which is a lie, this or that?

 

[mnsalem]

Hahahahaha get this crap, just got it in a new message to a ticket I had open:

Wait, what?! I thought in the last RFO report they said they had completed ALL node fixing and were now working on restoring? So which is a lie, this or that?

I got the same one, like, the EXACT wording .. so its another mass response, i was going to post it ... you beat me to it.

ya well, they're moving forward, a bit slow perhaps, but in the right direction ... i hope at least the backups are there. that's ALL i care about right now.

 

[mnsalem]

Just got a new email!

With the recent SolusVM exploits that have affected our company and others with a negative impact, many of our customers and us are not supportive of enabling public facing access to our SolusVM VPS CP as additional code could be exploitable. Let's not take a risk when it comes to security. At this time, we are releasing an alternative frontend solution to our customers to allow them to reboot, start, shut down, serial console, change root pass, or change hostname on their VPS. We hope to be making this more feature rich soon, however at the moment the only thing that you CANNOT do with this new frontend is: reinstall VPS, manage DNS entries, or create central backup. We are working on making these features available to you ASAP.

You can now access your virtual server controls at https://billing.chicagovps.net/clientarea.php?action=products . Select the service, and under the "Virtual Server Control" section you can manage multiple aspects of your VPS, including reboot, start, shut down, serial console, change root password, or change hostname.

No client's VPS data was leaked or accessed by a 3rd party during this hack. The hacker(s) did not directly access any VPS container or hypervisor, and simply used a SolusVM exploit to wipe out and cause damage to a certain number of VPS nodes. The intentions of the malicious hackers was cause mayhem within our company by wiping some of our servers. With this compromise, our SolusVM database was accessed by a third party. As such, there is a possibility that any passwords that were related with SolusVM could be at risk, for example your initial password you signed up with. For those clients VPS's that are now accessible and showing as an online state in the virtual server controls section in our client area, we urge that you immediately change your root password by clicking on the "Change Root Password" button.

Let it be clear that this compromise did not impact our client area in anyway, so any billing information, etc stored in our client area at billing.chicagovps.net is safe.

For good measure, please take a minute to change your client area password. Those who used the same SolusVM password as the client area should do this promptly. https://billing.chicagovps.net/clientarea.php?action=changepw

On a related note, rest assured we're making great progress in our recovery. A further update regarding this matter will be sent out later today.

We thank our customers for their continued support during this ordeal.

Regards,

ChicagoVPS Team

 

[drvelocity]

So it took them 5 f*ing days to just get a bunch of blank nodes up and running from scratch?  I could have done that singlehandedly in a day... ridiculous.

 

[Aldryic C'boas]

No client's VPS data was leaked or accessed by a 3rd party during this hack.

Umm...what?  The Solus master has direct access to every node - so yes, if the hacker knew of someone already at CVPS they wanted data of, it would be as simple as sending a command like tar zxvf /var/www/data.tgz /vz/private/<vservers.ctid>/ to the node, then grabbing the tarball of that VPS's data from the node's webserver.  To be quite honest, you should simply assume that someone has all of your VPS' contents from prior to the hack, and take the appropriate security precautions.

 

[sleddog]

Someone help me, I don't understand:

Your VPS goes down

You enquire to the company

Based on the response, or lack of, you make a decision

Wait...

Restore elsewhere from backups

CVPS is a budget company. Don't expect the world on a budget.

 

[mnsalem]

once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.

3 to 4 days????? what the fluff???



Remind me again, how did Magnificent Nick from RamNode do that in less than 2 days?????

 

[drvelocity]

CVPS 3-4 days = Probably not going to happen.  They just need more time to think about how to slowly equivocate until their customers just go away quietly.  Step 2: Restart the entire company under a new name!

 

[MannDude]

I'm sort of curious of the specs of their backup nodes. They've got 4 backup servers listed in Solus to handle backing up 109~ nodes, minus the Atlanta location. By the sounds of it, there is more than just Atlanta that wasn't backed up. LA, Chicago and Buffalo appear to be the only physical locations with backup servers, though they could of course be transferring data from Dallas to LA or to Chicago or something aswell I suppose.

 

[upsetcvps]

Umm...what?  The Solus master has direct access to every node - so yes, if the hacker knew of someone already at CVPS they wanted data of, it would be as simple as sending a command like tar zxvf /var/www/data.tgz /vz/private/<vservers.ctid>/ to the node, then grabbing the tarball of that VPS's data from the node's webserver.  To be quite honest, you should simply assume that someone has all of your VPS' contents from prior to the hack, and take the appropriate security precautions.

Not to mention solusvm has the ability to change passwords right?  So if one controls solusvm, ...

CVPS 3-4 days = Probably not going to happen.  They just need more time to think about how to slowly equivocate until their customers just go away quietly.  Step 2: Restart the entire company under a new name!

Yep.  Most users won't end up asking for a restore.  Those that do will just happen to be the ones that had corrupted or wiped backups.  How unlucky for them!

 

[Aldryic C'boas]

Not to mention solusvm has the ability to change passwords right?  So if one controls solusvm, ...

True enough.  What really should concern people is the database leak, though.  @ could confirm this... but wasn't the DB leak dated at least a day or more before the actual attack?  The guy could've grabbed any number of VPS dumps in the meantime before tearing everything up.

 

[jer]

Matthew || Staff  Saturday, June 22nd, 2013 (15:08)

At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.

We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.

You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).

Thank you again for your business and support.
---------------
Matthew
Support Guru

Jer S || Client  Saturday, June 22nd, 2013 (09:42)

Hey folks, still down. No ping.

I do not need a restore. I've looked at my rows in the leaked database, my container parameters don't seem broken.
----------------------------

Chris's response makes it sound like I want a restore. I don't. I don't know how to type it clearer.

So if you go in the Client Area to Services > Manage My VPS   - - it's broken.

Servers still down.