ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[jacobsta811]

Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.

 

[zulualpha]

Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.

So they're not sending out new root passwords after the DB leak? It's a good thing we live in such an honest world.  :unsure:

Was there any notification to you when your VPS came back up, or do we just have to keep trying indefinitely?

 

[upsetcvps]

One single server of the 10 of mine that are offline has come back up, albeit an empty install and I have no clue on God's green earth what the root password is to login either. I requested this server be restored from backup yesterday morning, so it's up but still no backup restored. Nor can I login to restore my own backups since I don't know what the root pass is haha.

About every hour they respond to another of my 20 tickets that are currently open pasting that same response in. What's funny is two hours ago that pasted response said they would have Solus back online shortly, well that's 2 hours ago ha ha.

They're bringing solus back up?  I think it's stupid for a provider like this to run software without access to the source.  Chris said no one listened to him last time and that's the reason this happened again.  So his solution is what?  Hope things don't happen in threes?

 

[jacobsta811]

I haven't gotten any new root passwords, but I agree they really should be resetting root passwords.

Edit: they have to bring back up a control panel sometime. Given their resources it seems to me it would be easier to just fix solus and the containers and use the labor of the customers to reimage, reset root passwords when they do, etc. Rather than try to do it themselves. Unfortunately that opens up the risk of another attack, but running with any decent number of customers basically requires the customers can do it themself.

 

[jfreak53]

Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.

The server in question that came up has no password available for login, it is only able to be logged in by someone using either serial console or SSH Auth, no password. So not possible considering SolusVM isn't up yet. Which means it's a fresh install and no backup placed yet.

Second, I never used nor do I EVER use Solus to change root password, I only ever use 'passwd' from root prompt, meaning solus has no clue what my root password was for any of my servers

 

[jfreak53]

They're bringing solus back up?  I think it's stupid for a provider like this to run software without access to the source.  Chris said no one listened to him last time and that's the reason this happened again.  So his solution is what?  Hope things don't happen in threes?

Of course, Solus finished their Security audit yesterday around 5 I think. Most providers brought solus online last night or early this morning. So as long as they have installed the most recent version patched they should be good. At least that's what I've been told by other providers.

 

[MannDude]

Woah, wait? They're not forcing a password reset on all customers? You've surely kidding. Well, if that's the case, yeah... change your password immediately when you get your VPS back. Who knows how many people has that DB leak now. It's bad enough your email and name is in it, even worse a password that you may or may not use in other places is in it too.

Change your password for anything that shares that.

Back in November when the DB was leaked the 1st time, Chris Fabozzi's password was in there for the admin login. He used the same password in other places, including Skype and LET, etc. Someone took his password from the DB leak, and had a bit of fun I believe. He should know the importance of informing his own customers to change their PWs.

 

[upsetcvps]

Woah, wait? They're not forcing a password reset on all customers? You've surely kidding. Well, if that's the case, yeah... change your password immediately when you get your VPS back. Who knows how many people has that DB leak now. It's bad enough your email and name is in it, even worse a password that you may or may not use in other places is in it too.

Change your password for anything that shares that.

Back in November when the DB was leaked the 1st time, Chris Fabozzi's password was in there for the admin login. He used the same password in other places, including Skype and LET, etc. Someone took his password from the DB leak, and had a bit of fun I believe. He should know the importance of informing his own customers to change their PWs.

 


The db also does not have my root password as I never used solusvm.  But I will be doing a fresh install anyway and restoring data from my own safe backups.

 

This is also why I use fake data for these cheap boxes.  Fake (well, not fake, but made just for the vps) e-mail, fake address, fake name.  The money is real; My data, well you cannot protect it apparently, so why should you have it.

 

[srichter]

Still can't ping my VPS on buf-17

 

[jfreak53]

I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.

 

[upsetcvps]

I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.

They're probably all on a plane to Brazil...

 

[SkylarM]

I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.

Hopefully for the sake of the clients, that isn't the case.

It's pretty interesting seeing something like this happen twice, and see how two totally different companies handle similar issues. Huge hats off to RamNode for dealing with it properly I must say.

 

[maounique]

In all honesty ramnode had much fewer nodes.

However, it is true CVPS could have handled it much better, especially since they already had this experience once.

 

[mikho]

In all honesty ramnode had much fewer nodes.

And not to forget, less vps / node.

 

[mnsalem]

Latest Report in right now:

Hey everyone, just a quick update. Since our last email, all nodes that were still affected have since been reinstalled. Right now we are working on installing the new VPS for each server and we are still making good progress.


When we reach final completion we will release another update.

Thank you all for your patience.

Regards

The ChicagoVPS Team

 

[maounique]

And not to forget, less vps / node.

That you cant know, could have been larger nodes, for example, in the end, it matters mostly the quantity of data, that is where the most delay should be.

Since Ramnode uses ssd on many nodes, i can figure the storage space is not that big, therefore restores should be faster.

But, again, even so, it took horibly long time to restore for CVPS. We are closing to a week now. Could have been faster, I think, but I do not have all the data so i could be wrong.

 

[jfreak53]

They say everything is back online but Pingdom still shows 4 Nodes down ha ha.

 

[mnsalem]

They say everything is back online but Pingdom still shows 4 Nodes down ha ha.

ya


But then you suspect if these were ever online! like, were they even online before the issues that occured? maybe they never were


Unless someone can confirm they were active.

 

[upsetcvps]

Hey everyone, just a quick update. Since our last email, all nodes that were still affected have since been reinstalled. Right now we are working on installing the new VPS for each server and we are still making good progress.


When we reach final completion we will release another update.


Thank you all for your patience.

Regards


The ChicagoVPS Team

I don't know what this means.  The last e-mail suggested that 2 hours from now everything would be restored.  Is that still the case...?  Sigh.

 

[Aldryic C'boas]

Since our last email, all nodes that were still affected have since been reinstalled.

This reads a bit like "if you don't have your own backups on these nodes, you're probably SOL".