ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]

[infinityhosting]

I just want a simple answer to when I will be in control of my vps again.

 

[drmike]

For those of you that are customers:

If you send me a private message I can look your account up and determine which server you are on.  That will help going forward to know what the issue is and independently do some self determination on resolution and patience.

Customers, if they knew the node, could look at say Pingdom and see if they are a node that is entirely offline or publicly clear having problems today.

Nodes with uptime issues today:

buf-vps14 - 3h 28min downtime

chi-vps13 - 0% uptime

chi-vps14 14h 32min downtime

chi-vps16  - 0% uptime

chi-vps17 - 0% uptime

chi-vps23 - 0% uptime

chi-vps24 - 0% uptime

chi-vps32 - 0% uptime

chi-vps40 - 3h 37min downtime

 

[Nth]

8230


This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.


We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.


Regards


The ChicagoVPS Team

Two servers per hour and we know from the DB dump they have around 100 servers, so sometime between two days and now.

buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?

 

[Chankster]

buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?

All DNS entries are as follows <Three Letter Location>-vps<Number>.chicagovps.net.  IE. CHI59 is chi-vps59.chicagovps.net. and yours would be chi-vps66.chicagovps.net

 

[mnsalem]

Two servers per hour and we know from the DB dump they have around 100 servers, so sometime between two days and now.

buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?

But then again, remember! not all of these 100 are down!

 

[funzie]

For those of you that are customers:

If you send me a private message I can look your account up and determine which server you are on.  That will help going forward to know what the issue is and independently do some self determination on resolution and patience.

Customers, if they knew the node, could look at say Pingdom and see if they are a node that is entirely offline or publicly clear having problems today.

Nodes with uptime issues today:

buf-vps14 - 3h 28min downtime

chi-vps13 - 0% uptime

chi-vps14 14h 32min downtime

chi-vps16  - 0% uptime

chi-vps17 - 0% uptime

chi-vps23 - 0% uptime

chi-vps24 - 0% uptime

chi-vps32 - 0% uptime

chi-vps40 - 3h 37min downtime

I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.

 

[drmike]

chi-vps66. Can you give me its IP?

 

IP looks like: 198.46.156.2

 

[drmike]

I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.

ATL-VPS2 was one of the nodes that particularly hard hit and they didn't have any backups in Atlanta.

Technically the server there has been rebuilt and is up.  But the VPS containers for customers is GONE.

 

[drmike]

Someone that is a provider / familiar with Solus answer this:

In nodes table, there is ftpbackup value, a 0 or 1.   Assuming 0 means ftpbackups are not enabled and a 1 means ftpbackups for the node are enabled.

Does that include client containers in that form of backup?

I see 26 nodes where ftpbackup is set as 0.

 

[upsetcvps]

Another useless update.

 

[jer]

I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.

That's what I was trying to say with my first post, but better said.

 

[drmike]

mysql> select nodeid,name from nodes where ftpbackup = 0;

+--------+-----------+

| nodeid | name      |

|      1 | localhost |

|     35 | chi22     |

|     21 | chi10     |

|     25 | chi13     |

|     31 | chi18     |

|     37 | chi24     |

|     39 | chi23     |

|     42 | chi27     |

|     79 | chissd1   |

|     48 | chi32     |

|     49 | chi33     |

|     57 | chi40     |

|     65 | chi47     |

|     68 | chi50     |

|     76 | chi51     |

|     80 | chi53     |

|    109 | atl1      |

|    110 | atl2      |

|    128 | atl3      |

|    131 | atl4      |

|    133 | atl5      |

|    138 | atl6      |

|    148 | nj1       |

|    149 | dfw1      |

|    150 | njkvm1    |

|    151 | chi70     |

+--------+-----------+


26 rows in set (0.00 sec)

 



ATL-VPS2 / ATL2 is on that list.  Those are the nodes where ftpbackups were not configured or were turned off. FTPBackups are the weekly backups that should be running automatically.

Someone asked about "centralbackups" manually ran by customers.  A provider with SolusVM experience will need to clarify where the centralbackups go if the ftpbackups are disabled.   I assume there is another setting / location for those --- hoping there is for those sitting and wondering.

 

[rds100]

The central backups go to special central backup server(s). But central backups are not made automatically, the user must go to SolusVM and make his own centralbackup.

By the way just because there is no ftp backup doesn't mean the node doesn't have backups. There are other ways to make node backups too, besiedes the solusvm ftpbackups feature.

 

[Nth]

All DNS entries are as follows <Three Letter Location>-vps<Number>.chicagovps.net.  IE. CHI59 is chi-vps59.chicagovps.net. and yours would be chi-vps66.chicagovps.net

Didn't even think to look it up by dns, thanks!

 

[drmike]

Another gotcha for folks still down:

If you are on a node that had extended downtime (5 hours+) in the past 3 days, you are on a node they most likely had to rebuild.

If you look at Pingdom, you can find the nodes in there with RED X's.   Those machines might be online, but the VPSes on them likely are not:

http://stats.pingdom.com/jzrszp4wfu79

I say that, because just received a few customer lookups and the servers show fine uptime today but customers are still offline (since Monday).

 

[leeboof]

Another gotcha for folks still down:

If you are on a node that had extended downtime (5 hours+) in the past 3 days, you are on a node they most likely had to rebuild.

If you look at Pingdom, you can find the nodes in there with RED X's.   Those machines might be online, but the VPSes on them likely are not:

http://stats.pingdom.com/jzrszp4wfu79

I say that, because just received a few customer lookups and the servers show fine uptime today but customers are still offline (since Monday).

Yeah my node looks online but VPS is still down as well. My VPS wasn't listed in the backup SQL file so not having much hope for a recovery...

Another thing that's weird is my server doesn't match the node I thought I was on from previous emails. The node it shows in the database I am now shows as never being down. 

Not sure which is correct.

 

[zero]

@XFS_Duke,  Simple as that. Just give it time.

time is over dude.

CVPS Mistakes;

1) Backup 

2) Disastery Plan

3) Customer Service and Communication

This is it ...

 

[Chankster]

Received this update on one of my tickets.

Hello,

We're getting pretty close to having this issue fully resolved. All nodes, and all customer containers will be back online in the next 24-36 hours. Further mass-update emails will be sent later today with additional updates. Thank you for your patience and understanding -- we appreciate your business very much.

---------------
Matthew
Support Guru

 

[zero]

I do not believe work everyting on 24-36 hour.

This is take gas in customers ...

How about backups ?

Whats The accuracy of backups ?

How day or week ago this backups ?

Whats the deleted vps customers status ?

How can give backups on lost vps's ?

How many vps up and runing now ?

Everything is blur ...

And chicago vps continues to persist communication ...

This is it CVPS not a institutional firm.

 

[zero]

We must a play a game.

If I CVPS Ceo what is my next movement ?  This is key question

If I CVPS Ceo : I hire a customer relationship person for communication ...