ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[CVPS_Chris]

cvps says it makes backups after the last incident. Turns out they have no clue. Some nodes don't have backups at all and they're not even sure how old the last backups were for the nodes that do have backups. If this was the first time this happened to you, cvps, ok maybe I cut you some slack. But it's not. And you should know better. You should learn from your mistakes in the past. You should know to inform your customers in a timely manner and keep them up to date. But you don't. Sure, I didn't expect much from you. But having used more expensive providers like Linode, I didn't think the premium paid for the name was worth what they provide (and it's not). But cvps, you managed to surprise me even with my low expectations of you. You could have handled this worse, but not by much.

Why are you saying false information? We have backups and know how old they are. As for this happening twice, its because no one listened to me when I said it was a Solus issue the first time. If I was listened to, maybe this would have been found months ago and it would have saved myself, Ramnode, and the other provider from what we are going through.

What do you expect in a 24 hour period? With all the problems we are dealing with its more important to get everyone back online that to write a response every hour saying "We are still working on it". Do you think I am sitting around eating a sandwich laughing at all of this? The answer is no, and this is a very serious matter.

I know what I say will not change your mind, but at least get your facts straight so you dont scare people that do not know better and will listen to you.

 

[Otakumatic]

Let CVPS do their work. God, some people are so impatient....

Also, the supposed "hackers", shut the fuck up.

/mytwocents

 

[texteditor]

As for this happening twice, its because no one listened to me when I said it was a Solus issue the first time. If I was listened to, maybe this would have been found months ago and it would have saved myself, Ramnode, and the other provider from what we are going through.

It's almost as if no one takes you seriously. Wonder why that is

 

[zero]

Anybody know what happend realy on cvps ?

or what now in there ?

 

[Chankster]

You couldn't be more wrong.  In a crisis situation it is still important to keep your customers informed. 

What do you expect in a 24 hour period? With all the problems we are dealing with its more important to get everyone back online that to write a response every hour saying "We are still working on it".

 

[WelltodoInformalCattle]

When customers log into their servers after this episode:

 

[drvelocity]

This is just an epic fail.  Obviously this company had no realistic backup/failsafe system in place for this kind of event despite already having this happen once before.

After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.

Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.

We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.

Regards

The ChicagoVPS Team

 

[zero]

@CVPS_Chris Excuse me but you must periodicly give information to customers. I'm your customer but I'm darkness in now. 

Which vps 's live or dead 

or what about the time frame I dont know.

Please give more information about this problem .... Please .....

 

[Nth]

Just a heads up for everyone waiting to get their VPS restored you have to open a ticket to have them to do it. I do see the logic in it as people with backups could restore thier servers faster than cvps can. Personally I don't feel like reuploading 20gigs with my slow upload speed. Even after this as long as they do manage to get my vps back up in a few days (I hope shorter time) I'd still consider it worth the 30 bucks I paid.

 

[CVPS_Chris]

@Zero, the recent email explains it all and should have all the fresh installs up within the next few hours. If you want us to try and restore from our backups, you need to open a ticket.

Obviously this company had no realistic backup/failsafe system in place for this kind of event despite already having this happen once before.

You could not be more wrong, we have backups for all nodes, we just simply cannot make them load any faster then they are, that is why I have decided to change the plan.

 

[Tux]

I still do not understand why people who have "important" data on a VPS do not keep own backups. Really. I even have backups of the most unimportant data. If one of my VPS provides goes down, it will take me a max. of 5 hours to be fully operational at another place. And I am just a "hobbyist"... Shame on all "professionals" for not having backups.

I totally agree. When one of my RamNode VPSes got wiped, I thankfully had a copy of the Minecraft world it hosted on another machine.

 

[zero]

@CVPS_Chris I need only data I dont care new vps or something. Which locations affected data loss and Whats the damage on there ?

 

[orizzle]

My machine is finally back up. Looks like the backup they restored was from about a week ago, though. Luckily I made my own backups. Switching to another VPS provider ASAP!

 

[CVPS_Chris]

restored was from about a week ago

It clearly states weekly backups.....

 

[zero]

@CVPS_Chris please check your private messages.

 

[xvtv]

Is buf-vps19 going to be up soon?

 

[zulualpha]

It clearly states weekly backups.....

So all the Buffalo nodes that aren't already up should be up in the next hour or so? 

Will new root passwords be emailed out to everyone? 

If customers choose to open a ticket and get their VPS restored from your backup individually, how much longer would that end up taking?

 

[drmike]

This can happen as they are probably in "all-hands-on-deck" mode, trying to recover from a disaster.

At last check there aren't many hands on board at ChicagoVPS.  The staff is laughably non existent for their user base:

select username from administrators;

+----------+

| username |

+----------+

| vpsadmin | 

| layotte  |

| fabocj40 |

| tleonard |

| adamng   |

| matthew 

 

6 accounts with some padding in there (i.e. fake accounts and a CC backdoor most likely).  layotte, tleonard and adamng are the three admins with tech know how.

 

Three admins to deal with how many customers?

 


select count(clientid) from clients;

+-----------------+

| count(clientid) |

+-----------------+

|            8025 |

+-----------------+

1 row in set (0.00 sec)

 


 

The total number of virtual servers active as of the hack?

 


select count(DISTINCT(vserverid)) from vservers where disabled = '0';;

+----------------------------+

| count(DISTINCT(vserverid)) |

+----------------------------+

|                       9357 |

+----------------------------+

1 row in set (0.07 sec)

 


Too many people put their "eggs" in an already "full fool" basket at ChicagoVPS.  You folks were buffalooed by those crazy low giveaway prices.

 

[JDiggity]

At last check there aren't many hands on board at ChicagoVPS.  The staff is laughably non existent for their user base:

select username from administrators;

+----------+

| username |

+----------+

| vpsadmin | 

| layotte  |

| fabocj40 |

| tleonard |

| adamng   |

| matthew 

 

6 accounts with some padding in there (i.e. fake accounts and a CC backdoor most likely).  layotte, tleonard and adamng are the three admins with tech know how.

 

Three admins to deal with how many customers?

 


select count(clientid) from clients;

+-----------------+

| count(clientid) |

+-----------------+

|            8025 |

+-----------------+

1 row in set (0.00 sec)

 


 

The total number of virtual servers active as of the hack?

 


select count(DISTINCT(vserverid)) from vservers where disabled = '0';;

+----------------------------+

| count(DISTINCT(vserverid)) |

+----------------------------+

|                       9357 |

+----------------------------+

1 row in set (0.07 sec)

 


Too many people put their "eggs" in an already "full fool" basket at ChicagoVPS.  You folks were buffalooed by those crazy low giveaway prices.

That is only 58 people per server if you figure 160 servers.  At least with Adam / Kevin has brought that down from the 100 per server under Jerimiah.

 

[drmike]

That is only 58 people per server if you figure 160 servers.  At least with Adam / Kevin has brought that down from the 100 per server under Jerimiah.

 

Well, it isn't that simple.  Fabozzi lied about the number of nodes then around the last hack and has lied prior to this hack.

select nodeid from nodes;

= a list of all nodes

High nodeid = 151.

But there aren't 151 nodes

select count(nodeid) from nodes;

+---------------+

| count(nodeid) |

+---------------+

|           109 |

+---------------+

1 row in set (0.00 sec)

 

 

Of those 109, 5 of them aren't VPS to customer nodes if I remember correctly (perhaps the backup servers?).  Comes down to 104 node servers.