ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[jfreak53]

Just a quick update to keep everyone in the loop. Within the next 6 hours we will have all servers reinstalled ready to restore backups on the remaining machines. Within 18 hours we expect to resume normal status and all customers to have running VPS' with data intact.
Thank you again for your patience!

Regards

The ChicagoVPS Team

So that email was sent last night at around midnight, 11:58PM to be exact. It says 6 hours all will be restored and then backups, but yet on Pingdom there are still 5 Chicago nodes offline ha ha







Wouldn't that have been like an hour ago ha ha? I might expect to see 1 offline still, but not the same 5.

 

[upsetcvps]

Well I'm glad they are at least giving time-frames now.  It's ok if they're a little off.  I wish I knew what exactly they are doing though.  Did they expect to have everyone up with a fresh vps in 6 hours (and how do I log in?) and then have the backups restored 12 hours after that?

Anyway, this is a slight improvement in communication.  Hopefully by the end of the day they will have everything sorted out.

 

[ramas]

Well I'm glad they are at least giving time-frames now.

As long as they're not giving random numbers just to calm their clients down. 2 days ago they said: "[SIZE=small]Our goal is to have everything 100% restored tomorrow."...[/SIZE]

Anyway, this is a slight improvement in communication.

Agreed.

They also mentioned they're dropping SolusVM completely and will look for an alternative. I'm wondering how long that's going to take and how're we suppose to control our VPS (once it's up and running) in a meantime.

 

[cvps_customer]

Learned my lesson on cheap vps's, will def hop over to linode. The responses i've gotten from Chris@chicagovps have been comical and contradict the emails that were sent out to all users. 

Corrupted backups? Testing your backups is Sys Admin 101, do these guys have any idea what they are doing? Going on 4 days downtime now, very glad I'm not relying on their service to host any sites of my own. If they didn't oversell their boxes they would easily be able to provide customers a VPS to use on one of the working nodes while working to restore data on the failed ones.

*EDIT* Looking back through the thread I noticed buffalo's post regarding ftp backups being disabled. This is absolutely ridiculous, such a joke of a business. 

 

[upsetcvps]

Learned my lesson on cheap vps's, will def hop over to linode. The responses i've gotten from Chris@chicagovps have been comical and contradict the emails that were sent out to all users. 


Corrupted backups? Testing your backups is Sys Admin 101, do these guys have any idea what they are doing? Going on 4 days downtime now, very glad I'm not relying on their service to host any sites of my own. If they didn't oversell their boxes they would easily be able to provide customers a VPS to use on one of the working nodes while working to restore data on the failed ones.

*EDIT* Looking back through the thread I noticed buffalo's post regarding ftp backups being disabled. This is absolutely ridiculous, such a joke of a business. 

That just means they weren't use the solusvm backup solution there.  They could have had their own.  Now they have definitely said some nodes don't have backups or were corrupted but not having ftpbackup through solusvm in particular isn't that terrible 

 

[cvps_customer]

That just means they weren't use the solusvm backup solution there.  They could have had their own.  Now they have definitely said some nodes don't have backups or were corrupted but not having ftpbackup through solusvm in particular isn't that terrible 

Thanks for the clarification, still hard to believe they don't have some type disaster recovery plan in place. 

 

[jer]

Just wanted to note that I'm on CHI-VPS10 - which reports up in pingdom, however my vps is not. I was told the reason for this is because some containers are damaged.

I have some 'container' experience. Both Virtualization Professional certs and the Virtualization Engineer cert from Parallels (not solusvm).

After looking at the CVPS dump for me and my container, I can say that the parameters for my container appear to be intact, and fine.

My node was done yesterday morning, I didn't ask for a restore, my container parameters don't appear damaged, and I'm still down.

I've had a long time suspicion that Chris leaves people down if he doesn't like them, and purposefully toys with them. I've confronted him before on it. I think this is one of those cases.

(continued)

 

[jer]

(Mods: I made this it's own post so it can be removed if needed without loosing the info in the first post.)

I've seen Chris call his customers "Idiots" on LET, had a long time suspicion that some employees were personas (now confirmed), Shinkle "terminated" my account (dropped my vps) for non-payment - on the day the payment was due (not the day after) - while I had a 2 day old open support ticket about how I couldn't get their payment portal to take my payment, that Shinkle had responded in. That was a while back. It's a good thing I know to keep good backups, imagine the impact to a regular Joe. Add that downtime to the previous Outage + this one.

Last month their payment system charged me twice. Chris gave me a credit (and avoided a charge back). So I'm 'pre-paid' for next month. It's one of the reasons I didn't switch providers 4 days ago.

There's an entire list, with tickets and conversations I've archived as far as 2 years back.

I put up with it because my sites aren't commercial, the network and throughput are good, and I'm getting a great deal.

I don't blame Chris for the SolusVM incident. I do blame Chris for how he's handled it. That does tack on to a growing list of how he's handled previous issues, and creates an opinion that Chris is not mature enough to run a company.

----------

If anyone can offer the same or nearly the same deal (or knows of a similar deal), is not a reseller of CVPS service and is not associated with a company Chris has had his hands in - I'll defect.

I currently have 2 IPv4s, 512M, 20G, Xen for $5.95. Network and throughput are important and prefer the host server located somewhere in the middle of the US. Please let me know via PM, not in the thread.

 

[upsetcvps]

I currently have 2 IPv4s, 512M, 20G, Zen for $5.95. Network and throughput are important and prefer the host server located somewhere in the middle of the US. Please let me know via PM, not in the thread.

I'm interested in alternative offers too.  Please post in thread!

 

[ramas]

I'm annoyed as much as you are. I have a numerous projects I'm working on and obviously I have my own servers/environments/etc that I manage. Even though I don't _really_ need a VPS, I figured heck if someone can take care of all the hardware/networking stuff for me and just give me a decent-usable environment for such a minimal price - I simply couldn't resist an offer and moved testing of some of my projects to ChicagoVPS.
Bet in general I really don't want to jump on this whole "bashing ChicagoVPS and even going personal" bandwagon. Sh*t happens I guess.

Now regarding the backups and some people claiming it's "clients fault" - I just don't agree. First of all, normally I wouldn't care less about the backups and such since that's not why *I* got my VPS for, EXCEPT for the fact that they list it as a service they provide. And it's not just some random advertisement wording strategy (which I would be fine with - everyone does it), but they have actually written it straight in their end-user agreement that I "signed" (stating: "fail-over protection and backup systems.").
In other words I'd have still got their VPS even if they clearly indicated they have no backups nor are responsible for my data; but since they agreed to provide me such service - obviously I'm quite disappointed.
For example "GreenValueHost" (from the top of my head) says they provide SSL certificates for their VPS. Great, and if I wouldn't get one after signing in I would be rightfully annoyed, no? Same with backups - if they say they have my backup - I expect them to have my backup (and there's a difference between "hackers damaged our backups" vs "we have no backups for some nodes").

If anyone can offer the same or nearly the same deal (or knows of a similar deal), is not a reseller of CVPS service and is not associated with a company Chris has had his hands in - I'll defect.

Unfortunately, I believe that's the point. As an old saying goes - you get what you paid for. If there was an alternative that has the same price and is "better", 99% of people would move there.

On a bright side with all this happening at least I learned what are the major aspects of a decent VPS/dedicated-server provider

 

[jer]

That's what I used my VPS for - a project dev environment. They're coming of age (the projects), but I don't make income from them so I don't want to go "full Linode" (although their service is outstanding IMO).

I don't think I can get a better price, the deal I have is great. I do think I can get a similar price for the same service.

 

[Amitz]

Look at http://Overzold.com (Prometeus) and http://www.waveride.at (Edis).

 

[Chankster]

So RFO #7 from 9hrs ago stated that all VPS's would be reinstalled and ready to restore backups in 6hrs.  Update I just got from a ticket stated that its going to be 12 hours before they hit that point.
 

We are getting closer to having all services restored -- at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time.
Once we've reached that point we will begin restoring customer containers for those who have requested it.
Doing so is a highly manual process so it will take quite some time; if you have your own backups you will be able to restore your data faster than we can.

Additionally we plan to re-enable SolusVM shortly as we are in the final stages of our security audit.

Thank you again for your patience as we work to get everything back online and running.

Josh
Support Guru

 

[leeboof]

So RFO #7 from 9hrs ago stated that all VPS's would be reinstalled and ready to restore backups in 6hrs.  Update I just got from a ticket stated that its going to be 12 hours before they hit that point.

12 hours from now or does that mean 3 more hours supposedly?

 

[Chankster]

12 hours from now or does that mean 3 more hours supposedly?

Sounds like 12 hours from now.  From ticket: "...at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time."

 

[mnsalem]

Sounds like 12 hours from now.  From ticket: "...at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time."

Most nodes are indeed back up according to Pingdom ... only 4 Chicago nodes left!


If they're still working on the 2 servers per hour rate ... then i think the nodes will be ready in the next couple of hours for operation!

 

[Chankster]

Most nodes are indeed back up according to Pingdom ... only 4 Chicago nodes left!


If they're still working on the 2 servers per hour rate ... then i think the nodes will be ready in the next couple of hours for operation!

Remember that nodes being up does not equal customer containers being up.  Pingdom shows that CHI59 has only had 3.75 hours of total down time but my container has been down since the morning of the 18th.

 

[upsetcvps]

I'm curious: has anyone who was down gotten there node back up since then?  With data or fresh?

 

[mnsalem]

Remember that nodes being up does not equal customer containers being up.  Pingdom shows that CHI59 has only had 3.75 hours of total down time but my container has been down since the morning of the 18th.

Ofcourse ... don't forget they need to get the host NODES up first THEN work on the CONTAINERS


But it is getting closer finally

 

[jfreak53]

I'm curious: has anyone who was down gotten there node back up since then?  With data or fresh?

One single server of the 10 of mine that are offline has come back up, albeit an empty install and I have no clue on God's green earth what the root password is to login either. I requested this server be restored from backup yesterday morning, so it's up but still no backup restored. Nor can I login to restore my own backups since I don't know what the root pass is haha.

About every hour they respond to another of my 20 tickets that are currently open pasting that same response in. What's funny is two hours ago that pasted response said they would have Solus back online shortly, well that's 2 hours ago ha ha.