amuck-landowner

ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

Status
Not open for further replies.

shovenose

New Member
Verified Provider
Every single BetterVPS sign up has been people running from ChicagoVPS. I can be proud of that. But for the price, ChicagoVPS provides the expected service. 

One of my ChicagoVPS-hosted nameservers now has a 1 day uptime but it's not erased and WHM is still accessible. I should probably change the password though.
 

xvtv

New Member
I have (had) one VPS with ChicagoVPS and another free with Host1Free, I'm glad I didn't bother to do backup between them...

They are still down...
 

infinityhosting

New Member
My vps is still down and I have not had a reply to my ticket. WAA-970698 I opened it 12 hours ago. I just want my vps back up so I can get back in business like everyone else.
 

drvelocity

New Member
I love how the owner of the company shows zero remorse for how much pain and agony his customers are suffering through and will continue to suffer through.  Not just counting the losses in sales from the downtime and potential client-loss, there will be thousands or tens of thousands of man hours required to rebuild all of these wiped VPS servers. 
 

upsetcvps

New Member
From http://chicagovps.net/about.html

What do you're current customers think of you?

hahahaha x2

It's like an 8-year-old wrote this page...

By the way Chris, what do your lucky customers win because of your 99.9% uptime guarantee?
 
Last edited by a moderator:

MannDude

Just a dude
vpsBoard Founder
Moderator
So, was there any official statement yet on what the cause of their hack was?

Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?
 

mnsalem

New Member
So, was there any official statement yet on what the cause of their hack was?

Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?
God knows.

Now i'm seeing more nodes going offline on Pingdom ... this is not good :/


Its supposed to go the other way around ...........
 

upsetcvps

New Member
So, was there any official statement yet on what the cause of their hack was?

Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?

Well Chris blamed solusvm (again) a few comments ago but who knows what that means.
 

MannDude

Just a dude
vpsBoard Founder
Moderator
God knows.

Now i'm seeing more nodes going offline on Pingdom ... this is not good :/


Its supposed to go the other way around ...........
Woah, can anyone who is a customer confirm if this is true or not: https://twitter.com/christruncer/status/347407782057742336

"So, @ChicagoVPS decided that restoring customer VPSs from their backups is “too time consuming” and is instead telling people to start over."
Could be some data on the backup nodes ( I think there was 4 of them ) could have been wiped to?
 

upsetcvps

New Member
Woah, can anyone who is a customer confirm if this is true or not: https://twitter.com/christruncer/status/347407782057742336

Could be some data on the backup nodes ( I think there was 4 of them ) could have been wiped to?
yeah the tweet is true.  Here's their latest e-mail (it was posted earlier here but I'll repost):

 After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.

Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.

We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.

Regards

The ChicagoVPS Team
Mind you, there are *still* people without a vps of any kind, so I can't imagine what the fuck they are doing.
 
Last edited by a moderator:

zulualpha

New Member
I'm still without my two VPSs in Buffalo. I've got my own backups, and since they're not restoring backups now I don't know what's taking so long to get the nodes up
 

Mun

Never Forget
Cameron Munroe (Munroenet)

After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.

Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.

We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.

Regards

The ChicagoVPS Team
 

jacobsta811

New Member
Got the same "we don't really have any backups, or least any automated restore". The only one of my 4 nodes back up at all today was atlanta, and that was up, I changed the password, and shut it down because apt-get update failed. Atlanta was back up again, with the *old* password later, so they reimaged it twice (no tickets in at all right now). Shut it down again because apt-get update and apt-get anything was still failing. God only knows what they are doing. Buffalo, Los Angeles, Chicago all still down. All of these are the 2GB/$40/yr plans.

I do note that however or whatever they are "overselling" does *NOT* matter. All that matters is actual performance of my VPS and any issues it has. While they were running performance was satisfactory and I had no issues. Clearly they have giant issues outside of normal operation, but the amount of overselling/overprovisioning does not matter unless it causes an *actual* impact.
 
Last edited by a moderator:

jacobsta811

New Member
Clearly they don't, or it isn't setup right, or they have no backups.  Else we'd be restored by now, 36 hours later, wouldn't we ?
 

Nick_A

Provider of the year (2014)
I don't know about central restore, but regular old ftp backups have to be done one by one in SolusVM. This is why it took me 24 hours to do a few hundred VPSs. Only one can be restored per node at a time.
 

drmike

100% Tier-1 Gogent
Folks unsure of the status of their VPS should submit tickets to CVPS directly.  I know their staff is insanely flooded and probably will have a good wait time for a response.

Folks are chattering a bit on Facebook: https://www.facebook.com/chicagovps/posts/683328475017314

If you lost data, it is your fault.  You are responsible for your own data.  Provider backups aren't sufficient ever and often take eons to have restored.  They are a last resort sort of thing.

If you feel slighted by the uptime guarantee, file with them for a credit.  ChicagoVPS is offering at least a month credit to those impacted.

apt-get problems = more problems.   Post the error and we'll help you debug.

Overselling = extreme.  Problem with the overselling more than anything is overselling relative to staff resources.  3 admins doing 8 hour shifts each means 7 day work weeks and folks always on call.   A company with 8k-9k customers should be better staffed - at minimum 5-7 technical administrators.   When you buy a VPS that costs more than $7 for 2GB, having redundancy, security and staff should be places that cause the bottom line to increase.  Think about that when shopping for your new provider and quit being a cheapskate solely.

Finally, it doesn't seem this hack will be disclosed or determined what occurred.  The one in November went the same way.   Finger pointing at Solus then and this time.   Since the SolusLabs folks failed to do anything in public about the accusations in November/February, we can only assume Solus had bugs and themselves couldn't figure out the vulnerability.   

ChicagoVPS didn't hack their system and destroy data.  A criminal did that.  CVPS is a victim as much as the customers who have lost their data and services.
 

jacobsta811

New Member
apt-get was throwing errors that it couldn't resolve the hostnames of the mirrors. Could be a DNS issue with whatever DNS they have setup by default, or it could be an issue with the setup in the restored slice. Normally it works out of the box in a reimage. I could have spent time trying to solve it, but it was an indication to me that it isn't ready for me to screw with, so I just shut it down (and obviously I can't restart it without solusVM or opening a ticket).

RE:Overselling. Overselling vs # of support staff is a big problem as we can see now, I agree. But people here were also complaining about overselling on specific nodes, and I disagree about that, for any provider - as long as the performance remains adequate for what I need 100% of the time, it isn't and shouldn't be my or your concern how they have the server provisioned/sold.

I am not worried about lost data, as I maybe lost 2 blog posts or something. More time will be lost setting up the boxes again than anything. The bigger issue is having a place to *put* my data. Most of the cheap hosts use SolusVM so I am pretty much out of luck right now. I'm thinking I probably will get a node at South Bend VPS and setup there. I had a full mirrored setup using IP failover, unison, and MySQL replication with geographic separation of the servers in Atlanta and Buffalo, but both were on ChicagoVPS. Clearly an issue, but it seems like I would have needed not only two providers but two different *control panels* to really be safe.

Edit to add: I run my mail server out of Linode, but they are kind of pricy and the disk space is really small on the reasonably priced options for use for my personal websites/blogs/etc that are not really that critical.
 
Last edited by a moderator:

upsetcvps

New Member
The bigger issue is having a place to *put* my data. Most of the cheap hosts use SolusVM so I am pretty much out of luck right now. I'm thinking I probably will get a node at South Bend VPS and setup there. I had a full mirrored setup using IP failover, unison, and MySQL replication with geographic separation of the servers in Atlanta and Buffalo, but both were on ChicagoVPS. Clearly an issue, but it seems like I would have needed not only two providers but two different *control panels* to really be safe.
my other vps is over at buyvm and it works well enough for me.  It's also cheap but at least they do use stallion instead of solusvm
 
Status
Not open for further replies.
Top
amuck-landowner