# ChicagoVPS / CVPS Hacked. New SolusVM exploit? (Content Restored) [PT. 1/2]



## Magiobiwan

Well, looks like CVPS has also fallen victim to the latest SolusVM Exploit. Given their CP page anyways. 



Take a look for yourself! https://manage.chicagovps.net:5656/

Someone posted this on LET, but nobody had posted here. YET.

*EDIT: TO COMMENT ON THIS THREAD PLEASE VISIT PART 2 HERE: *http://vpsboard.com/topic/758-chicagovps-cvps-hacked-new-solusvm-exploit-pt-22/

-MannDude


----------



## Magiobiwan

Well, looks like CVPS has also fallen victim to the latest SolusVM Exploit. Given their CP page anyways. 



Take a look for yourself! https://manage.chicagovps.net:5656/

Someone posted this on LET, but nobody had posted here. YET.


----------



## Francisco

Is it just a DB dump?

You should censor the image

Francisco


----------



## drmike

Umm link doesn't work on CVPS and the 4chan file has been pulled:

This file is no longer available because of claim by  _4shared Support Team_.


----------



## earl

Lately it seems this is a tough business to be in..


----------



## drmike

ChicagoVPS is a total crap heap.

This is the second hack since November.

Yeah, there is a full database dump floating around too it seems.

Did anyone see what was in that database passwords file?


----------



## Magiobiwan

I didn't grab it myself, but I hear (from thread on LET) it's the same stuff as was in the RamNode one. Names, Emails, Client IDs, Hashed PW, Original Root PW, IPs, Hostnames, OS, Virt Type, etc.


----------



## drmike

Oh look, Kevin is indeed Adam Ng:

(12,'adamng','50f1ba0c5fe73f19bbf09cc728f2526e57910c23','[email protected]','Adam','Ng','Active',1,1354156121,'1','sysinfo:true,susvps:true,rlsinfo:true|clusterstats:true,userstats:true,clientact:true,updates:true,sysmess:true|','control:true,sett

ings:true|usage:true,info:true,note:true|','nodeinfo:true,nodebuttons:true,nodegraphs:true|','')


----------



## MannDude

0_o


----------



## mpkossen

buffalooed said:


> ChicagoVPS is a total crap heap.
> 
> This is the second hack since November.
> 
> Yeah, there is a full database dump floating around too it seems.
> 
> Did anyone see what was in that database passwords file?


Could also be due to a SolusVM exploit.

Several people on LET confirm their information is in the file. I'm just assuming it's a dump of the customer table or maybe even a full dump.


----------



## HalfEatenPie

Yikes!  Not another one!


----------



## Amitz

Maybe #gothacked is the new #winning?


----------



## drmike

mpkossen said:


> Could also be due to a SolusVM exploit.
> 
> Several people on LET confirm their information is in the file. I'm just assuming it's a dump of the customer table or maybe even a full dump.


It's a full SolusVM dump.  Same sort of dump as the November #failing


----------



## MannDude

So does this DB have all the same info as their last one did? If so, that's very worrying. If it's a full DB dump of SolusVM then I'd assume it has emails, passwords (hashed, can be easily unhashed), IP addresses, etc, etc.

Not good at all. Was this Robert Clarke too? Haha.


----------



## peterw

Post on LEB:



> Lol:
> 
> 
> Attention providers:
> In roughly 12 hours I will be disclosing 3 zero day vulnerabilities on solusvm.
> I suggest you take backups fast or else hackers will abuse this. I have tried contacting solusvm to fix it however I have been ignored.
> They stated the vuln is “not important at this time”
> Good luck.
> 
> June 17, 2013  2:36 pm


----------



## MannDude

buffalooed said:


> Oh look, Kevin is indeed Adam Ng:
> 
> (12,'adamng','50f1ba0c5fe73f19bbf09cc728f2526e57910c23','[email protected]','Adam','Ng','Active',1,1354156121,'1','sysinfo:true,susvps:true,rlsinfo:true|clusterstats:true,userstats:true,clientact:true,updates:true,sysmess:true|','control:true,sett
> 
> ings:true|usage:true,info:true,note:true|','nodeinfo:true,nodebuttons:true,nodegraphs:true|','')


Ha, been saying it all along. No one cares. Adam Ng IS Kevin Hillstrand. No one would admit to anything. The kid just turned 18 a month or so ago.


----------



## drmike

MannDude said:


> If it's a full DB dump of SolusVM then I'd assume it has emails, passwords (hashed, can be easily unhashed), IP addresses, etc, etc.


 

Yep it's the same type of info.  Obviously, current data though.


----------



## drmike

MannDude said:


> Ha, been saying it all along. No one cares. Adam Ng IS Kevin Hillstrand. No one would admit to anything. The kid just turned 18 a month or so ago.



Yeah Holestained is Adam Ng.      Yep, he was a minor while employed by CVPS and URpad.

Yes, Adam and Chris lied again about who Hillstrand was.

Did WHT ban Hillstrand yet?  They should.


----------



## MannDude

buffalooed said:


> Yep it's the same type of info.  Obviously, current data though.


Yikes. Wonder who else has been hit who hasn't been made public yet.


----------



## drmike

MannDude said:


> Yikes. Wonder who else has been hit who hasn't been made public yet.


 

Just heard someone say they have 3 VPSes down at CVPS now.

Anyone else have things down over there?


----------



## drmike

From LET:



> ihatetonyy Member
> 
> 
> 7:57AM edited 7:57AM
> 
> 
> 
> 
> 
> In Chicago, not sure what node:
> 
> -bash-4.1# ls
> Segmentation fault
> -bash-4.1# uptime
> Segmentation fault
> -bash-4.1# ls
> Segmentation fault
> -bash-4.1# ps ax
> Segmentation fault
> -bash-4.1# ls
> Segmentation fault
> -bash-4.1# uptime
> Segmentation fault
> -bash-4.1#


----------



## MannDude

buffalooed said:


> Just heard someone say they have 3 VPSes down at CVPS now.
> 
> Anyone else have things down over there?


I don't know anyone who has a VPS with them, LET probably has more who do than from here. Quick Twitter search shows a few people complaining about VPSes being down: https://twitter.com/search?q=chicagovps&src=typd

After the last incident they setup backup servers so hopefully everyone who is down has a backup made.


----------



## maounique

Guys, maybe cvps and chris in particular are unsavoury and liars, however, this is a very serious stuff, if there are 3 more vulnerabilities and maybe a full "crate" of those, we are seriously screwed and have to replace Solus.

Francisco's Stallion looks very good now, in fact too good. Coincidence ?


----------



## mpkossen

My VPS in NJ appears to be down.


----------



## drmike

Mao said:


> Francisco's Stallion looks very good now, in fact too good. Coincidence ?


 

Didn't Stallion originally have roots in Solus?  (not to get off track)

I've never heard of Fran bringing Stallion to market as a product for other providers.  Might not be a bad idea though.


----------



## Francisco

Mao said:


> Guys, maybe cvps and chris in particular are unsavoury and liars, however, this is a very serious stuff, if there are 3 more vulnerabilities and maybe a full "crate" of those, we are seriously screwed and have to replace Solus.
> 
> Francisco's Stallion looks very good now, in fact too good. Coincidence ?


It isn't for sale.

I will likely give a few hosts a free copy but I haven't decided who.

Francisco


----------



## ashworth

Yep right now we're down. Posted here:

http://www.webhostingtalk.com/showthread.php?p=8730252

Friggin' lame. This is getting old.


----------



## Magiobiwan

I doubt BuyVM would do this. Too risky. Anyways, if you're a provider and HAVE NOT taken SolusVM down, YOU SHOULD!


----------



## MannDude

Does anyone have a CVPS VPS that is ONLINE?

CVPS_Chris is reading this thread but I'd imagine he's too busy to respond right now.

Best of luck to everyone and hopefully their backups can be restored quickly.


----------



## drmike

Magiobiwan said:


> if you're a provider and HAVE NOT taken SolusVM down, YOU SHOULD!


 

To reiterate, someone has posted that they have at least 3 hacks for SolusVM that are unknown and have given a 12 hour timeline for providers to lock things down.

Their rationale is that they reported the vulnerabilities to Solus and Solus hasn't done squat.


----------



## mpkossen

MannDude said:


> Does anyone have a CVPS VPS that is ONLINE?
> 
> CVPS_Chris is reading this thread but I'd imagine he's too busy to respond right now.
> 
> Best of luck to everyone and hopefully their backups can be restored quickly.


On Skype he told me they are aware of the situation, so I guess they're working on it.


----------



## ihatetonyy

MannDude said:


> Does anyone have a CVPS VPS that is ONLINE?
> 
> CVPS_Chris is reading this thread but I'd imagine he's too busy to respond right now.
> 
> Best of luck to everyone and hopefully their backups can be restored quickly.


Yes. One in LA that hasn't had any data fuckery yet and has reasonable load again.




Code:


[[email protected] ~]# uptime
 12:17:16 up 27 days,  8:08,  1 user,  load average: 0.03, 0.13, 0.15


----------



## ashworth

Chris Fabozzi just updated in a ticket that they're working on it and will release a statement shortly (after I went crazy in a ticket).

Glad to know someone's listening at least, and that the CEO can take a moment to respond.


----------



## Magiobiwan

And read this thread too?


----------



## D. Strout

Just got an e-mail from Versatile IT stating that they have shut down SolusVM for a while - basically until after the vulnerabilities are released and the damage can be assessed.


----------



## MannDude

Magiobiwan said:


> And read this thread too?


I'd imagine he's got the thread open on any forum talking about it hitting F5 between cussing and answering tickets. I'd be trying to stay as in the loop as possible too.


----------



## ashworth

Yeah, no kidding. Pretty much what I'm doing right now


----------



## drmike

What a shame, I PM'd @CVPS_Chris prior to this going public telling him I wasn't sure that Kevin was doing a good job checking on system compromises since in fact Kevin doesn't exist.

Ghost employees really make lousy employees.


----------



## Francisco

buffalooed said:


> Didn't Stallion originally have roots in Solus?  (not to get off track)
> 
> I've never heard of Fran bringing Stallion to market as a product for other providers.  Might not be a bad idea though.


We originally used solus but broke away after they pulled BS with us and modifying a few pages.

Stallion 1 still uses a *very* old solusvm skin though.

Francisco


----------



## Kris

My 3 are down. No funky messages even. Good call on the central backups. How doesn't Solus have some sort of mod security type rules in-between the web server?

The developers clearly can't secure the program, going to need predictive Mod Security / OWASP rules in between their *shit* software.


----------



## drmike

Lots of folks are getting segmentation faults.

Usually when you see those, the OS has been destroyed.   I suspect nodes are getting wiped.


----------



## D. Strout

buffalooed said:


> To reiterate, someone has posted that they have at least 3 hacks for SolusVM that are unknown and have given a 12 hour timeline for providers to lock things down.
> 
> Their rationale is that they reported the vulnerabilities to Solus and Solus hasn't done squat.


Is there any evidence backing up their claim? I mean aside from the ChicagoVPS hack?


----------



## drmike

D. Strout said:


> Is there any evidence backing up their claim? I mean aside from the ChicagoVPS hack?


 

Ahh, nope.  But real providers would be mighty foolish not to take strong precautionary actions to prevent being a victim.

If the exploits carry the impact the other known one does, it is a full system compromise.


----------



## peppr

I have one in their LA node which is still online, however quite slow. SSH is responding though.


----------



## MannDude

Edited the title to be more relevant. This may be a new exploit from what I've been reading and from that LET thread.

Scary stuff.

I'd be turning off my SolusVM master if I were a provider to be safe. If clients complain, just tell them why it's down.

If SolusVM was truly warned about a new exploit, and did not act on it, then shame on them.


----------



## D. Strout

buffalooed said:


> Ahh, nope.  But real providers would be mighty foolish not to take strong precautionary actions to prevent being a victim.


Definitely agree, but I'm just trying to figure out how much chance that this is just some guy making idle threats.


----------



## trewq

MannDude said:


> If SolusVM was truly warned about a new exploit, and did not act on it, then shame on them.


If this is the case I will not be using SolusVM any longer.


----------



## D. Strout

trewq said:


> If this is the case I will not be using SolusVM any longer.


...If only it were that easy. There is no really _good_, no-compromise alternative. SolusVM comes with its own compromises, but _generally_ it works. And people are familiar with it. It won't be easy for any provider to just wave bye-bye to something so well-established.


----------



## drmike

D. Strout said:


> There is no really good, no-compromise alternative. SolusVM comes with its own compromises, but generally it works.


 

As usual, I am not a provider.  I fail to see how complicated a panel could be.   Minimal, finite number of things it needs to do.

Certainly are alternatives and certainly should be renewed interest in extending APIs to create your own panel like Backupsy has on top of Proxmox.

Too many complacent folks doing the same as the other providers.  When something breaks, it is mass failure across many businesses.

I miss the days when people built their own solutions for most things.


----------



## Magiobiwan

I suspect that following this series of incidents, there's going to be a large increase in home-grown panel solutions.


----------



## peterw

This is insane:



> @BradND said: Pulled our solus, seriously suggest everyone else does also





> @Patrik: Done the same.





> @Magiobiwan: I just pulled BlueVM's SolusVM down as well.





> @Maounique: Yes, did too, shut down the machine just to make sure this is not a backdoor left by someone using the old exploit, checked before but you can never be sure, if the 3 new exploits are jokes, we will just reinstall, but so far looks grim.





> @john: We've also taken our SolusVM offline now. Better safe than sorry.





> @trewq: Versatile IT's SolusVM is now shutdown.





> @AnthonySmith: Shut the solusvm masters down completely to avoid being hit, this is just messed up.


----------



## Magiobiwan

Again, better safe than compromised, with your DB dumped and made public and your nodes rm -rf --no-preserve-root /'ed.


----------



## john

What @Magiobiwan said. Let's hope any potential exploits can be confirmed soon as real or not so we can restore SolusVM access.


----------



## MartinD

Someone who has access or knowledge of these so-called vulns needs to let SolusVM know so they can be investigated.


----------



## MartinD

Someone who has access or knowledge of these so-called vulns needs to let SolusVM know so they can be investigated.


----------



## drmike

Liam over at LET has read the Kevin = Adam info and he....


 




> Liam Administrator
> 
> 
> 8:58AM
> 
> 
> 
> 
> 
> In light of all the evidence given to us, we have gone ahead and changed Adam's username.



About damn time.  Permabanning  the account would be the right thing to do.


----------



## MannDude

buffalooed said:


> Liam over at LET has read the Kevin = Adam info and he....
> 
> 
> 
> 
> 
> 
> 
> About damn time.  Permabanning  the account would be the right thing to do.


That's hilarious. I've been saying it all along.


----------



## peterw

@CVPS_Adam



> We had patched the centralbackup.php almost immediately on Sunday morning, and per a post on LEB ( http://www.lowendbox.com/blog/solusvm-vulnerability/#comment-121070 ) - there may be more problems with SolusVM. We've been told that other code besides the originally exploited centralbackup.php also utilizes the PHP exec function, and I personally do not believe it is safe as of right now for any provider to have their SolusVM install on right now until we have a better understanding of things. SolusVM's management staff are engaged and working closely with us.
> 
> Further updates will be posted shortly as we work through this ordeal.


----------



## drmike

> BradND Member
> 
> 
> 9:21AM
> 
> 
> 
> 
> 
> @CVPS_Adam I'm not sure anyone cares about you being hacked, why did you lie about being kevin?
> 
> 
> Bradley, NodeDeploy


----------



## D. Strout

Hasn't anyone decrypted the source? Couldn't they then run a search for dumb execs?


----------



## drmike

> peppr Member
> 
> 
> 
> 9:25AM
> 
> 
> 
> 
> 
> @CVPS_Adam Finally Adam, you are the 0day to be happily patched today. Welcome to LET


----------



## concerto49

Have shut down Solus also in light of things for now.


----------



## drmike

Someone somewhere said CVPS has 8 servers that are messed up.  In their world that would be 400-800 VPS containers.

They are unsure if Adam/Kevin patched things yesterday...  Ba-dumb!


----------



## XFS_Duke

Seems to be 9 and steady...


----------



## netnub

This is what happens when you don't have good security.

Now, where shall I release more zero day vulnerabilities ?

If you search hard enough you'll find I released a zero-day on the admin login page which allowed the use of eval() which you could easily compromise server from it. I've only released it to show how SHITTY solusvm is.

They don't properly clean the submitted data.


----------



## vanarp

netnub said:


> If you search hard enough you'll find I released a zero-day on the admin login page which allowed the use of eval() which you could easily compromise server from it. I've only released it to show how SHITTY solusvm is.


 
When did you release it?


----------



## netnub

vanarp said:


> When did you release it?


As I stated in a lowendbox post, I was going to release it in 8 hours from when I posted yesterday, and I did.


----------



## Jack

netnub said:


> As I stated in a lowendbox post, I was going to release it in 8 hours from when I posted yesterday, and I did.



Want some advice?

Stop wasting your time on a control panel as that will soon turn into


----------



## netnub

Jack said:


> Want some advice?
> 
> Stop wasting your time on a control panel as that will soon turn into


Want some advice?



doesn't scare me.

What are they going to do, sue me for helping them get better security? I already attempted to contact them, they failed to do anything.


----------



## FHN-Eric

Good thing I'm running Virtualizor. Even more of a reason to not switch to solus. Now if only Virtualizor could prevent httpd from going down every 4 hours or so.


----------



## Jack

netnub said:


> Want some advice?
> 
> 
> 
> doesn't scare me.
> 
> What are they going to do, sue me for helping them get better security? I already attempted to contact them, they failed to do anything.


You know what you've done to CVPS?

If you don't have Solus on your arse, I'm sure Chris is going to want words at a minimum with you.


----------



## Jack

netnub said:


> Want some advice?
> 
> 
> 
> doesn't scare me.
> 
> What are they going to do, sue me for helping them get better security? I already attempted to contact them, they failed to do anything.


You know what you've done to CVPS?

If you don't have Solus on your arse, I'm sure Chris is going to want words at a minimum with you.


----------



## netnub

FHN-Eric said:


> Good thing I'm running Virtualizor. Even more of a reason to not switch to solus. Now if only Virtualizor could prevent httpd from going down every 4 hours or so.


Have you considered running a script to see if httpd is down, if it is start it up?


----------



## netnub

FHN-Eric said:


> Good thing I'm running Virtualizor. Even more of a reason to not switch to solus. Now if only Virtualizor could prevent httpd from going down every 4 hours or so.


Have you considered running a script to see if httpd is down, if it is start it up?


----------



## MartinD

netnub said:


> What are they going to do, sue me for helping them get better security? I already attempted to contact them, they failed to do anything.


I *STRONGLY *suggest you contact them again, i.e. immediately, with whatever information you have.


----------



## MartinD

netnub said:


> What are they going to do, sue me for helping them get better security? I already attempted to contact them, they failed to do anything.


I *STRONGLY *suggest you contact them again, i.e. immediately, with whatever information you have.


----------



## netnub

Jack said:


> You know what you've done to CVPS?
> 
> If you don't have Solus on your arse, I'm sure Chris is going to want words at a minimum with you.


What I've done? Its nothing illegal. I never hacked CVPS.


----------



## netnub

Jack said:


> You know what you've done to CVPS?
> 
> If you don't have Solus on your arse, I'm sure Chris is going to want words at a minimum with you.


What I've done? Its nothing illegal. I never hacked CVPS.


----------



## netnub

MartinD said:


> I *STRONGLY *suggest you contact them again, i.e. immediately, with whatever information you have.


Maybe we should stop twittling our thumbs and do something about it?

The following code has been obfuscated for solusvms security:


----------



## netnub

MartinD said:


> I *STRONGLY *suggest you contact them again, i.e. immediately, with whatever information you have.


Maybe we should stop twittling our thumbs and do something about it?

The following code has been obfuscated for solusvms security:


----------



## FHN-Eric

MartinD said:


> I *STRONGLY *suggest you contact them again, i.e. immediately, with whatever information you have.


Wouldnt hold my breath on him doing that.


----------



## mitgib

netnub said:


> Maybe we should stop twittling our thumbs and do something about it?
> 
> The following code has been obfuscated for solusvms security:


And if someone restricts access to only the admin api and only lets the WHMCS plugin access?  Leaving solus down makes for angry villagers


----------



## MartinD

FHN-Eric said:


> Wouldnt hold my breath on him doing that.


Simple fact is, if he knows what the problem is and doesn't bother telling the developers then ultimately he's harming himself. No-one will trust him, no-one will want is control panel and no-one will want to provide him with services.

Look at this logically - he is holding every single provider who uses SolusVM to ransom. Do you really want someone like that hanging around?


----------



## notFound

I have reported the vulns I have been made aware of to SolusVM by that little birdy, it doesn't really take a genuis to figure any of them out once you have access to the un-encoded version of SolusVM. I'm not holding my breath on SolusVM responding.

(_Yes, and I'm sure I've just given away to everyone who I actually am. ;-)_)


----------



## D. Strout

So... have I missed something? When/where are/have been these vulnerabilities being/been posted?


----------



## FHN-Eric

notFound said:


> I have reported the vulns I have been made aware of to SolusVM by that little birdy, it doesn't really take a genuis to figure any of them out once you have access to the un-encoded version of SolusVM. I'm not holding my breath on SolusVM responding.
> 
> (_Yes, and I'm sure I've just given away to everyone who I actually am. ;-)_)


I just told solus about that as well, and linked them to this thread. Now to hope they get it to managent as requested


----------



## notFound

I didn't just tell about the thread, I have examples to them as they requested (they responded fast, let's see if they respond fast again). 

EDIT: They're "looking into it now."


----------



## netnub

FHN-Eric said:


> I just told solus about that as well, and linked them to this thread. Now to hope they get it to managent as requested


Nice IP address, you may want to hide that.


----------



## FHN-Eric

netnub said:


> Nice IP address, you may want to hide that.


Your point? I got nothing to hide, its just an ip.


----------



## netsat

@netnub

You should have your fu***** balls cut off

/Johnny


----------



## netnub

I should have my balls cut off? That's a bit harsh. Oh well, kids will be kids.


----------



## netsat

The only kid around here is you. You enjoy making trouble for providers who are just trying to make a living - not to mention all the users.

/Johnny


----------



## vanarp

This is interesting...

80 user(s) are reading this topic
26 members, 53 guests, 0 anonymous users


----------



## MannDude

Now now. Let's try to keep this on topic.

Those of you with servers at CVPS, have they sent out client wide emails yet informing their clients to reset passwords? Any official word from them?

I don't see anything on their Twitter or Facebook, so curious if they're taking this instance more serious than the last one. Are all the VPSes back online now?


----------



## uidzer0

I don't see what the big deal is. Security researchers find bugs and exploits all day long. Most of the time they reach out to the vendor and either work a deal out with them on a release date of the bug/exploit or if they don't hear anything, they just release it as a 0day. He said he reached out to solus initially and never heard back from them, then as this thread blew up all of sudden solus is interested in hearing what's wrong with their product.

I would much rather hear about an issue with something I'm using and know the problem is there rather then not hear anything at all and just have my shit pwned all day long.


----------



## FHN-Eric

[email protected] said:


> @netnub
> 
> You should have your fu***** balls cut off
> 
> /Johnny


Wow a bit harsh. Sure Curtis found an exploit, but I highly doubt he is stupid enough to hack CVPS, of course I've been wrong before.


----------



## MartinD

Personally, I'd like to see proof that he contacts Solus and that they ignored what he had to say.

It makes no sense for any developer to do that.


----------



## uidzer0

Full-disclosure. Love it or Hate it.


----------



## vld

MartinD said:


> Personally, I'd like to see proof that he contacts Solus and that they ignored what he had to say.


Personally, I'd like to see proof that anything that this curtis guy said he did is true. All of the stuff he posted till now is pretty much BS, including the parts of the Solus code he posted.


----------



## netnub

uidzer0 said:


> Full-disclosure. Love it or Hate it.


Exactly


----------



## FHN-Eric

vld said:


> Personally, I'd like to see proof that anything that this curtis guy said he did is true. All of the stuff he posted till now is pretty much BS, including the parts of the Solus code he posted.


Couldnt have said it better myself.


----------



## wlanboy

vanarp said:


> This is interesting...
> 
> 80 user(s) are reading this topic
> 26 members, 53 guests, 0 anonymous users


Yup. 54 members, 104 guests are on board now.


----------



## MartinD

vld said:


> Personally, I'd like to see proof that anything that this curtis guy said he did is true. All of the stuff he posted till now is pretty much BS, including the parts of the Solus code he posted.


Well, he claims to have the full, unencoded version of Solus... yet he posts a snippet of code that's encoded. :blink:


----------



## netnub

MartinD said:


> Well, he claims to have the full, unencoded version of Solus... yet he posts a snippet of code that's encoded. :blink:


Yes, I stated in a above post which you removed that "for solusvm security, here is obfuscated version" which is obfuscated functions/variables.


----------



## MartinD

Ah, their security.

Still not contacted them I presume? You know, for their and everyone else's security?


----------



## FHN-Eric

Well thats good, the ticket is under managent review  amazing at what requesting managent to see it can do. I'll let you know what managent says when they reply.


----------



## netnub

I keep posting proof I hold the source code, however staff remove my posts.


----------



## Craig0ry

@netnub

I can't really see you proving you've found anything. This topic is beyond a joke, looks like CVPS didn't patch quick enough like they stated. So now everyone is turning off SolusVM all because these idiots lied to cover there ass and this NetNub is making false claims!

Read SolusVM blog - it seems to be all rumours


----------



## MannDude

Netnub has posted source. He's sent a PM to myself with the source code as well.

I don't think it's wise to post these publicly, it's best SolusVM reviews and patches first. We don't want more hosts being impacted. I've seen some of the hits this forum has gotten from Google and judging my certain search phrases that are bringing them here I can guarantee if the source was posted on here it'd be used for anything but good.


----------



## Jono20201

MannDude said:


> Now now. Let's try to keep this on topic.
> 
> Those of you with servers at CVPS, have they sent out client wide emails yet informing their clients to reset passwords? Any official word from them?
> 
> I don't see anything on their Twitter or Facebook, so curious if they're taking this instance more serious than the last one. Are all the VPSes back online now?


Last email I have from them is an Invoice.


----------



## CVPS_Chris

Craig0ry said:


> looks like CVPS didn't patch quick enough like they stated.


We most certainly did, please do not call us a liar. Just like the first hack in November Solus said nothing was wrong on their end, and then Ramnode happened and this happened.

I even asked Phil if its a possibility that hack from RamNode was the same in November and he said YES. Of course they will call it rumors so they dont look bad and the terrible product they released.


----------



## notFound

MannDude said:


> Netnub has posted source. He's sent a PM to myself with the source code as well.
> 
> I don't think it's wise to post these publicly, it's best SolusVM reviews and patches first. We don't want more hosts being impacted. I've seen some of the hits this forum has gotten from Google and judging my certain search phrases that are bringing them here I can guarantee if the source was posted on here it'd be used for anything but good.


I have seen the source to and can confirm that there are some examples of really bad coding (which I have sent to SolusVM and hence those "blocks of code" refered to here http://blog.soluslabs.com/2013/06/18/statement-regarding-current-security-rumours/ which I'm sure others have notified them of too). Also, I do agree it's unwise to release source here, once it gets in the wrong hands.. Well I'm sure it already is in the wrong hands already but we don't want more.


----------



## JDiggity

notFound=JoePie?

netstat = JohnnyDbag?


----------



## Craig0ry

CVPS_Chris said:


> Of course they will call it rumors so they dont look bad and the terrible product they released.


Pretty much same as what you've done. Blamed it on SolusVM? How come i know websites still running SolusVM and they've not been hacked? Bit mysterious that.........


----------



## netnub

Ill be posting more later. I have work to do.


----------



## MannDude

CVPS_Chris, since you are here:

1.) Why have you not informed your clients yet? C'mon man. You should have learned from the other hack. Don't leave your clients in the dark. Just send out a mass email like all the other hosts have and explain the situation.

2.) Still going to deny the Adam = Kevin thing? I had a shit-ton of proof but was waiting to post it when (or if ever) prompted for it. Looks like I don't need to post my proof anymore.

3.) What has been impacted aside from the DB being leaked? What sort of issues are you guys experiencing? Data loss? Corrupt files? What's going on?


----------



## notFound

24khost said:


> notFound=JoePie?
> 
> netstat = JohnnyDbag?


Nope, you've got the complete wrong end of the stick. ;-)

Doesn't take a genius to figure out who I am or netstat is.


----------



## Navarr

Hey guys, CVPS customer here, just checking in.

I've got a buffalo server and it's completely up and running.  

SSH, HTTP, nothing seems affected but control panel. - No Contact from CVPS


[email protected]:~$ uptime
 09:32:20 up 30 days, 18:32,  2 users,  load average: 0.04, 0.05, 0.00
It's a shame that passwords were leaked, in what looks to be a SHA1 hash - which speaks loads to the security of the system (why are they not using an actual secure password system?  

Anyone who's ANYONE in the PHP world knows to use Bcrypt instead of SHA1), which would at the very least prevent rainbow tables!


----------



## netnub

I know exactly who notfound is. I have no idea who netstat is but I got a good idea who.


----------



## FHN-Eric

SolusVM management told me they are looking into it.


----------



## FHN-Eric

SolusVM management told me they are looking into it.


----------



## ashworth

MannDude said:


> Now now. Let's try to keep this on topic.
> 
> Those of you with servers at CVPS, have they sent out client wide emails yet informing their clients to reset passwords? Any official word from them?
> 
> I don't see anything on their Twitter or Facebook, so curious if they're taking this instance more serious than the last one. Are all the VPSes back online now?


Just this at 2:26 AM PST in a ticket:



> SolusVM was hacked, and a user started deleting data. We are not sure what the total overall damage is yet.
> If you are offline, its because the data was deleted, not that we turned them off. If you are in any location other than Atlanta, we have backups
> 
> Regards
> 
> ---------------
> 
> Chris Fabozzi
> 
> CEO / Director of Operations


----------



## MCH-Phil

MannDude said:


> 1.) Why have you not informed your clients yet? C'mon man. You should have learned from the other hack. Don't leave your clients in the dark. Just send out a mass email like all the other hosts have and explain the situation.


Not emailing your customers is just bad.  Great job *@**CVPS_Chris*!


----------



## Mun

My question is why don't providers have nginx block things that aren't needed by clients?


----------



## rds100

notFound said:


> I have seen the source to and can confirm that there are some examples of really bad coding (which I have sent to SolusVM and hence those "blocks of code" refered to here http://blog.soluslabs.com/2013/06/18/statement-regarding-current-security-rumours/ which I'm sure others have notified them of too). Also, I do agree it's unwise to release source here, once it gets in the wrong hands.. Well I'm sure it already is in the wrong hands already but we don't want more.


If you've seen the code just post it somewhere and let everyone else see it, goddamnit. That's the best thing you can do and that's the only thing that could help secure the damn code.


----------



## MannDude

Navarr said:


> Hey guys, CVPS customer here, just checking in.
> 
> I've got a buffalo server and it's completely up and running.
> 
> SSH, HTTP, nothing seems affected but control panel. - No Contact from CVPS
> 
> 
> [email protected]:~$ uptime
> 09:32:20 up 30 days, 18:32,  2 users,  load average: 0.04, 0.05, 0.00
> It's a shame that passwords were leaked, in what looks to be a SHA1 hash - which speaks loads to the security of the system (why are they not using an actual secure password system?
> 
> Anyone who's ANYONE in the PHP world knows to use Bcrypt instead of SHA1), which would at the very least prevent rainbow tables!


Thanks for the report, glad to know it's up and running and not _everyone_ was impacted with downed servers / data loss. I do think the fact they've yet to make any public announcements to warn their customers about their information being leaked is very, very worrying. I hope they do that soon.

Also, welcome to vpsBoard. I hope you stick around and enjoy your stay!



ashworth said:


> Just this at 2:26 AM PST in a ticket:


Thanks for the update! Good to see they're around and actively responding to tickets.


----------



## JDiggity

notFound said:


> Nope, you've got the complete wrong end of the stick. ;-)
> 
> Doesn't take a genius to figure out who I am or netstat is.


I am not sure who netstat is but signed his post /johnny which I figured since Eric is here JohnnyDbag can't be far behind.

notFound not sure haven't read alot of your posts, so can't tell who you are based on this thread.


----------



## concerto49

24khost said:


> I am not sure who netstat is but signed his post /johnny which I figured since Eric is here JohnnyDbag can't be far behind.
> 
> notFound not sure haven't read alot of your posts, so can't tell who you are based on this thread.


notFound can be found on LET as a mod. Hint hint.


----------



## JDiggity

ahhh that says it all.


----------



## MannDude

concerto49 said:


> notFound can be found on LET as a mod. Hint hint.


I know who he is, but not sure if he wants to me tell or not. There are a lot of members with aliases on here from LET... =]


----------



## FHN-Eric

24khost said:


> I am not sure who netstat is but signed his post /johnny which I figured since Eric is here JohnnyDbag can't be far behind.
> 
> notFound not sure haven't read alot of your posts, so can't tell who you are based on this thread.


Just to point out, I joined before him. Why does he keep following me? 24khost, I got to webhostrally.com before you did.


----------



## FHN-Eric

MannDude said:


> I know who he is, but not sure if he wants to me tell or not. There are a lot of members with aliases on here from LET... =]


This might be a wild guess, but could it be Liam?


----------



## GIANT_CRAB

So much drama, I like.


----------



## mnsalem

Navarr said:


> Hey guys, CVPS customer here, just checking in.
> 
> I've got a buffalo server and it's completely up and running.
> 
> SSH, HTTP, nothing seems affected but control panel. - No Contact from CVPS
> 
> 
> [email protected]:~$ uptime
> 09:32:20 up 30 days, 18:32,  2 users,  load average: 0.04, 0.05, 0.00
> It's a shame that passwords were leaked, in what looks to be a SHA1 hash - which speaks loads to the security of the system (why are they not using an actual secure password system?
> 
> Anyone who's ANYONE in the PHP world knows to use Bcrypt instead of SHA1), which would at the very least prevent rainbow tables!


Seems like you're one of them lucky ones ... in Buffalo here as well and my VPS is down .. Maybe I'm on a different node .. who knows? What i know for sure is that I'm moving out the moment its back up from the backup.


my MISTAKE is not looking up CVPS online before ordering.


----------



## MannDude

mnsalem said:


> Seems like you're one of them lucky ones ... in Buffalo here as well and my VPS is down .. Maybe I'm on a different node .. who knows? What i know for sure is that I'm moving out the moment its back up from the backup.
> 
> 
> my MISTAKE is not looking up CVPS online before ordering.


Welcome to vpsBoard as well. Seems a few new members have been joining when searching about the CVPS hack it seems? What node are you on?

If you've got a VPS up or down, I think it's beneficial to post what node you're on so other members on the same node can comment if they're up/down too.


----------



## FHN-Eric

mnsalem said:


> Seems like you're one of them lucky ones ... in Buffalo here as well and my VPS is down .. Maybe I'm on a different node .. who knows? What i know for sure is that I'm moving out the moment its back up from the backup.
> 
> 
> my MISTAKE is not looking up CVPS online before ordering.


If your looking for a new provider, 24khost, NodeDeploy, WSWD, and SonicVPS are good providers. Hope CVPS did good backups on a regular bases, if the backup is corrupt that wont be usefull in restoring data


----------



## saliq

Anyone know where I can get this database file ? I would like to see if I`m in it... 

Im in NY DC and no downtime so far, everything is working..

10:14:39 up 30 days, 21:47,  3 users,  load average: 0.00, 0.00, 0.00


----------



## netnub

Loved or hated but never ignored.


I contacted solusvm in ticket my ticket was deleted. Will upload pictures later when I get off iPhone.


----------



## MannDude

FHN-Eric said:


> If your looking for a new provider, 24khost, NodeDeploy, WSWD, and SonicVPS are good providers. Hope CVPS did good backups on a regular bases, if the backup is corrupt that wont be usefull in restoring data


After they got hacked in November they added backup nodes. Not sure how many or how often backups of VMs are made. Not sure if it's automatic or an additional feature customers have to activate themselves or what. If you're in Atlanta, and your vps data is gone, it's gone. Chris or Adam or someone said on LET all locations are backed up other than Atlanta. So who knows?


----------



## mnsalem

MannDude said:


> Welcome to vpsBoard as well. Seems a few new members have been joining when searching about the CVPS hack it seems? What node are you on?
> 
> If you've got a VPS up or down, I think it's beneficial to post what node you're on so other members on the same node can comment if they're up/down too.


Thanks!

How do I find out? Is there a way to find out which is it without Solus (which is offline until now)?


If their IP Addresses are split on nodes, then I'm on the one with the 192.227.xxx.xxx subnet _*(UPDATE: buf19 node)*_ (if there's a risk posting this feel free to remove it)


I'm trying to check via the client area for any information on that ... unsuccessful so far. the billing site is vey slow at the moment.


----------



## mnsalem

MannDude said:


> After they got hacked in November they added backup nodes. Not sure how many or how often backups of VMs are made. Not sure if it's automatic or an additional feature customers have to activate themselves or what. If you're in Atlanta, and your vps data is gone, it's gone. Chris or Adam or someone said on LET all locations are backed up other than Atlanta. So who knows?


I remember asking about the backups when i signed up. i just fug the ticket up and that was their response then:

_"*Weekly backups* are done on each node automatically so in the event of a node crash we can easily restore the data, however this is rare since we are RAID10 protected."_


----------



## FHN-Eric

Yes, but did CVPS verify the integrity of the backups when they were done? If not how do you know they arnt currupt?


----------



## ashworth

mnsalem said:


> What i know for sure is that I'm moving out the moment its back up from the backup.


I haven't received any word from CVPS on whether or not "backups" are, in fact, available to use after what happened.

Do you know something I don't? Extremely anxious about that specific item. Emailed them this morning an hour ago, but no reply on that one just yet.


----------



## Mun

ashworth said:


> I haven't received any word from CVPS on whether or not "backups" are, in fact, available to use after what happened.
> 
> Do you know something I don't? Extremely anxious about that specific item. Emailed them this morning an hour ago, but no reply on that one just yet.


Just that this has happened before, and last time backups were pretty much non-existent.

Outline of there responses: http://www.lowendhelp.com/chicagovps-net-attacked-archive-of-emails/ for the last one and not the current attack.

Mun


----------



## mnsalem

ashworth said:


> I haven't received any word from CVPS on whether or not "backups" are, in fact, available to use after what happened.
> 
> Do you know something I don't? Extremely anxious about that specific item. Emailed them this morning an hour ago, but no reply on that one just yet.


Well, i'm also still waiting on my ticket for more than an hour as well ..

what i noted above in my previous reply was 3 months ago ... when i asked them _"about the weekly backup system that is offered with the deal"_ (3GB RAM VPS Promo 3 months back)


Backups specifically this time mean alot to me, as the last backup was downloaded a month ago (i was travelling and i was stupid enough to forget to activate the cron to get the weekly backup i prepare).


----------



## mnpeep

https://www.youtube.com/watch?v=CnaVoTfkqa8

Pretty much sums it all up...


----------



## Lanarchy

CVPS

CHI - up and accessible

ATL - 1 down, 1 up but not accessible via SSH or anything and replies to ping

NY - down

LA - down


----------



## ashworth

I'm in hell. This sucks.


----------



## MCH-Phil

ashworth said:


> I'm in hell. This sucks.


It's not just you


----------



## SeriesN

Wow! Just God damn Wow!


----------



## Cloudrck

mnpeep said:


> https://www.youtube.com/watch?v=CnaVoTfkqa8
> 
> Pretty much sums it all up...


This has always been my viewpoint of any online presence.


----------



## cvps_customer

Another CVPS customer here, only communication i've received is the blanket email response to my ticket at 9 a.m. this morning saying a statement is coming. I have 2 hosts completely down, 1 in ATL and 1 in NY. 

 I've been trying to track down the DB dump but havn't been able to find it yet, just wanted to see what info of mine is out there.


----------



## mnsalem

ashworth said:


> I'm in hell. This sucks.


Same here mate ... Same here.



cvps_customer said:


> Another CVPS customer here, only communication i've received is the blanket email response to my ticket at 9 a.m. this morning saying a statement is coming. I have 2 hosts completely down, 1 in ATL and 1 in NY.
> 
> I've been trying to track down the DB dump but havn't been able to find it yet, just wanted to see what info of mine is out there.


From what i'm seeing ... EVERYONE's credentials and data associated to it was compromised! In additionh to the lost data from the servers that is ..


First name, Last name, Email, User ID, Hashed Password, if you have a VPS on CVPS, your data is in the list.


----------



## XFS_Duke

I'm pretty sure that if the DB dump is out there, then all of your information is in it. Please make sure you don't slack and start changing passwords... Not sure if they'll do anything to your account or not as it seems that the people are just after SolusVM, but... don't take any chances...


----------



## Asim

I have been passing around the link on my twitter so probably thats because of this (@asimzeeshan)


----------



## ashworth

cvps_customer said:


> I've been trying to track down the DB dump but havn't been able to find it yet, just wanted to see what info of mine is out there.


If you can PM me on the board with your email address, I'll confirm if you're on it. You probably are.

Found it on a mirror a second ago, but I won't post it here for fear of violating a board rules or something.


----------



## upsetcvps

Lanarchy said:


> CVPS
> 
> CHI - up and accessible
> 
> ATL - 1 down, 1 up but not accessible via SSH or anything and replies to ping
> 
> NY - down
> 
> LA - down


well I can't access my chicago vps in CHI...

Actually, never mind.  I am positive I ordered a CHI vps but geo ip points to NY.  HAHA, another job well done cvps!


----------



## XFS_Duke

Each location has multiple nodes.. Some were affected while some others weren't...

Some peoples accounts might be online and some might not due to them being on different nodes.

Chris has said that they have backups though they might be a week old at the latest for everything but Atlanta. Chris, if you could confirm that here, that would be awesome and probably set some minds at ease...

Once again, if you have a ChicagoVPS account, CHANGE YOUR PASSWORDS IF YOU USE THE SAME ONE.


----------



## redjersey

mnsalem said:


> Same here mate ... Same here.
> 
> From what i'm seeing ... EVERYONE's credentials and data associated to it was compromised! In additionh to the lost data from the servers that is ..
> 
> 
> First name, Last name, Email, User ID, Hashed Password, if you have a VPS on CVPS, your data is in the list.


first whmcs and now chicagovps. I guess I will receive tons of spams from now on.

luckily my chicagovps is still online (NY/Buffalo) so not all servers are down


----------



## shovenose

My ChicagoVPS VPS in Chicago is up, unaffected. No communications from them though.

ShoveHost SolusVM, well, I shut the entire server down after a mass email.


----------



## mikho

MartinD said:


> Simple fact is, if he knows what the problem is and doesn't bother telling the developers then ultimately he's harming himself. No-one will trust him, no-one will want is control panel and no-one will want to provide him with services.
> 
> 
> Do you really want someone like that hanging around?


I think you can remove "will" from all the sentences. No-one should trust this name changer, trying to get some easy money from the people over at LET. Not once but multiple times.


----------



## netnub

SolusVM decided to ignore my ticket; I opened one a hour or so ago, they bumped it to management review, then they deleted the ticket. I have pictures on my iPhone as I submitted it from there.


I guess its time.


----------



## shovenose

Sure, go ahead. Our SolusVM is off and as are most of the providers that are any good around here. Release that shit and be done with it.


----------



## MartinD

netnub said:


> SolusVM decided to ignore my ticket; I opened one a hour or so ago, they bumped it to management review, then they deleted the ticket. I have pictures on my iPhone as I submitted it from there.
> 
> 
> I guess its time.


You're talking utter crap. I would LOVE to see this proof from your 'iPhone'.


----------



## MannDude

netnub said:


> SolusVM decided to ignore my ticket; I opened one a hour or so ago, they bumped it to management review, then they deleted the ticket. I have pictures on my iPhone as I submitted it from there.
> 
> 
> I guess its time.


If you've got proof that you submitted a ticket, as in a screenshot or something, then please do post.


----------



## Tactical

I wanna see 2. I about to start making some ye gar bombs. Anyone want one? Cause the poop is gonna hit the ceiling! lol


----------



## mnsalem

Latest email still BEING sent out to users, as reported on LET (not everyone received this yet)



> Around 3am Eastern Standat Time (EST) today, there was a security breach, due to a vulnerability in SolusVM that allowed a command line to be run to dump the ChicagoVPS SolusVM client database and attempt to delete all data from our nodes. Our staff is working tirelessly to get everything back online, along working with SolusVM to address the root issue and no furthur impact is expected.
> 
> Now what does this mean for the customer? All passwords should be changed, this includes passwords for SolusVM control panel and your VPS. This data leak does not include billing information or credit card information. Thus far we are having great success in getting nodes back online with no data loss, however, there are a few that were not recoverable and will be restored using our offsite backups.
> 
> Once the situation is 100% complete and back to normal we will send another email out. We understand the sevarity and importance to get everything back online quickly. With that in mind, please try to refrain from opening a ticket or replying to an old one as it only slows us down even more. We are doing our best, and hope to have this fully resolved within 24 hours.
> 
> Thank you for your patience and understanding.
> 
> Regards
> 
> Your ChicagoVPS Team


----------



## Cloudrck

MartinD said:


> You're talking utter crap. I would LOVE to see this proof from your 'iPhone'.


Why would you assume he's lying, because you don't like what he said? After SolusVM has been exposed for their bad programming techniques, is this really that hard to believe?


----------



## MartinD

Because I've seen evidence to the contrary, that's why.


----------



## XFS_Duke

SgtZinn said:


> I wanna see 2. I about to start making some ye gar bombs. Anyone want one? Cause the poop is gonna hit the ceiling! lol


The poop already hit the ceiling... lol


----------



## sv01

waiting for people from CVPS give more details regarding hack and backup files. Don't tell (again) it's unknown exploit and you wont report that


----------



## XFS_Duke

MartinD said:


> Because I've seen evidence to the contrary, that's why.


Can you elaborate on that a bit? To set peoples minds at a little ease


----------



## XFS_Duke

@sv01,

You'll need to wait a bit longer


----------



## Aldryic C'boas

> Can you elaborate on that a bit? To set peoples minds at a little ease


He means that the source of the claim is untrustworthy, and should not be taken at face value just because of the situation at hand.


----------



## MartinD

XFS_Duke said:


> Can you elaborate on that a bit? To set peoples minds at a little ease


There's nothing I can say to ease the minds of users. All I can say is I have seen evidence that contradicts what CurtisG is claiming that the ticket was deleted and that he received no reply.

Yes, I do know Phill. Yes, I have spoken to him a number of times during all of this crap which is why I've been more keen to see the 'evidence' of such exploits.


----------



## MartinD

Aldryic C said:


> He means that the source of the claim is untrustworthy, and should not be taken at face value just because of the situation at hand.


^ this. Exactly this.


----------



## mnsalem

> Around 3am Eastern Standat Time (EST) today, there was a security breach, due to a vulnerability in SolusVM that allowed a command line to be run to dump the ChicagoVPS SolusVM client database and attempt to delete all data from our nodes. Our staff is working tirelessly to get everything back online, along working with SolusVM to address the root issue and no furthur impact is expected.
> 
> Now what does this mean for the customer? All passwords should be changed, this includes passwords for SolusVM control panel and your VPS. This data leak does not include billing information or credit card information. Thus far we are having great success in getting nodes back online with no data loss, however, there are a few that were not recoverable and will be restored using our offsite backups.
> 
> Once the situation is 100% complete and back to normal we will send another email out. We understand the sevarity and importance to get everything back online quickly. With that in mind, please try to refrain from opening a ticket or replying to an old one as it only slows us down even more. We are doing our best, and hope to have this fully resolved within 24 hours.
> 
> Thank you for your patience and understanding.
> 
> Regards
> 
> Your ChicagoVPS Team


COPIED over from LET.


----------



## XFS_Duke

MartinD said:


> ^ this. Exactly this.


Ok cool. What I meant by ease peoples minds is that this dude said SolusVM ignored him... Kinda seemed bad in the eyes of a customer/provider that they would do that knowing the severity of the vulnerabilities... Atleast we have someone with more knowledge then the rest of us


----------



## Tactical

They should put more emphasis on the passwords part like this *CHANGE YOUR PASSWORDS!. *


----------



## MartinD

XFS_Duke said:


> Ok cool. What I meant by ease peoples minds is that this dude said SolusVM ignored him... Kinda seemed bad in the eyes of a customer/provider that they would do that knowing the severity of the vulnerabilities... Atleast we have someone with more knowledge then the rest of us


I'm saying the opposite, that he wasn't ignored


----------



## Kris

Who wrote that?

furthur? Standat? sevarity?

Was that seriously emailed out?


----------



## netnub

MartinD said:


> Because I've seen evidence to the contrary, that's why.


Challenge.... accepted.


----------



## ashworth

Chris just replied to my ticket saying:



> If your VPS was not in Atlanta, then yes we have backups


----------



## mnsalem

Kris said:


> Who wrote that?
> 
> furthur? Standat? sevarity?
> 
> Was that seriously emailed out?


Well, CVPS_Chris mentioned on LET that an email was sent out, i got nothing in my inbox personally .. but this was shared by another LET member who says got the email.

they did mention as well they're looking for another Panel to replace Solus


----------



## XFS_Duke

On the email thing... it takes time... might not be in your inbox... Might be in junk mail or whatever...

For the pictures that netnub posted, well, i'm not sure what to say about that... Does the ticket not show up in your account? Since we cannot see the URL or ticket ID in the second screenshot then we can't really tell...


----------



## ashworth

mnsalem said:


> i got nothing in my inbox personally


 

I didn't get an email, either. Read it here on this board.


----------



## netnub

XFS_Duke said:


> On the email thing... it takes time... might not be in your inbox... Might be in junk mail or whatever...
> 
> For the pictures that netnub posted, well, i'm not sure what to say about that... Does the ticket not show up in your account? Since we cannot see the URL or ticket ID in the second screenshot then we can't really tell...


No, it doesn't show up in the account. It was deleted.


----------



## SVMPhill

netnub said:


> SolusVM decided to ignore my ticket; I opened one a hour or so ago, they bumped it to management review, then they deleted the ticket. I have pictures on my iPhone as I submitted it from there.
> 
> 
> I guess its time.


The ticket was not deleted. It was replied too.

#GBL-110307 - Contact Form: Your clock has expired
Submitted at 18/06/2013 14:47 and replied to at 18/06/2013 15:10


----------



## ashworth

If a mod/admin wouldn't mind chiming in, I'm wondering if it's all cool with you guys for me to post the link to the data for customers who are concerned with what exactly leaked. I have a working link, but don't want to post unless I have a mod's blessing first.


----------



## mnsalem

ashworth said:


> I didn't get an email, either. Read it here on this board.


Its up in the client area now ..

https://billing.chicagovps.net/announcements.php?id=3


----------



## ashworth

If a mod/admin wouldn't mind chiming in, I'm wondering if it's all cool with you guys for me to post the link to the data for customers who are concerned with what exactly leaked. I have a working link, but don't want to post unless I have a mod's blessing first.


----------



## netnub

ashworth said:


> If a mod/admin wouldn't mind chiming in, I'm wondering if it's all cool with you guys for me to post the link to the data for customers who are concerned with what exactly leaked. I have a working link, but don't want to post unless I have a mod's blessing first.


I got a working link also, its been uploaded like 50 times already.


----------



## ashworth

netnub said:


> its been uploaded like 50 times already


 

Well yeah, but the original mirror is down. Just trying to be respectful in the forum, as I'm new. Don't be smug.


----------



## Craig0ry

This kid with all the claims is rather boring now.

You need a job, rather than making false claims. You must live a boring life :lol:


----------



## crspyjohn

Geez another hack on CVPS... twice in less than a year


----------



## MannDude

Do *not* post the link to the DB here, please.

If you're a customer, just rest assured your data is in the leak. There is no possible way your data would not be in the leak.

If you're genuinely curious and absolutely must have it, search harder. Otherwise just accept the fate that your data is leaked, and you need to change your passwords immediately, and the password of any other account that may be the same as the one in the leak (IE your email, or wordpress login, etc)


----------



## FHN-Eric

netnub said:


> I got a working link also, its been uploaded like 50 times already.


Gee. I wonder how long it took him to find one. Considering the fact that he has provided no evidence, and the fact that he can decode ioncube. Doesnt take a genious to know that Curtis knows a working link, of course you stated it in the open.


----------



## ashworth

MannDude said:


> Do *not* post the link to the DB here, please.
> 
> If you're a customer, just rest assured your data is in the leak. There is no possible way your data would not be in the leak.
> 
> If you're genuinely curious and absolutely must have it, search harder. Otherwise just accept the fate that your data is leaked, and you need to change your passwords immediately, and the password of any other account that may be the same as the one in the leak (IE your email, or wordpress login, etc)


Thanks for confirming. Appreciate that...new I was legit in my hesitation!   

On that note, please halt the PM's to my account. Getting emailed like crazy. Thanks much.


----------



## mnpeep

MannDude said:


> Do *not* post the link to the DB here, please.
> 
> If you're a customer, just rest assured your data is in the leak. There is no possible way your data would not be in the leak.
> 
> If you're genuinely curious and absolutely must have it, search harder. Otherwise just accept the fate that your data is leaked, and you need to change your passwords immediately, and the password of any other account that may be the same as the one in the leak (IE your email, or wordpress login, etc)


Not sure how, but my password was translated to password, and I know for a fact that my password on CVPS is not password. Hmm...


----------



## XFS_Duke

ashworth said:


> Thanks for confirming. Appreciate that...new I was legit in my hesitation!
> 
> On that note, please halt the PM's to my account. Getting emailed like crazy. Thanks much.


I don't see a reason why people should have it anyways. If you were a ChicagoVPS customers, your information is out there. There is no need for people to download this. And anyone with a working link should report the link ASAP to the hoster so that it gets deleted. People that do this type of stuff obviously have no life at all and all you're doing when you share the link is promoting their dumb shit.


----------



## epaslv

SVM_Phill said:


> The ticket was not deleted. It was replied too.
> 
> #GBL-110307 - Contact Form: Your clock has expired
> Submitted at 18/06/2013 14:47 and replied to at 18/06/2013 15:10


This is great, spins bullshit for hours about the ticket being deleted when he cant use his iphone...


----------



## XFS_Duke

Well myself, I like seeing proof. Anyone can sign up as SVM_Phill and reply to a thread with random numbers... Admins, do yall have proof that this was indeed Phillip Bandelow from SolusLabs?


----------



## netnub

SVM_Phill said:


> The ticket was not deleted. It was replied too.
> 
> 
> #GBL-110307 - Contact Form: Your clock has expired
> 
> 
> Submitted at 18/06/2013 14:47 and replied to at 18/06/2013 15:10


Interesting how it doesn't exist then, eh?


----------



## XFS_Duke

That is 2 totally different support tickets... netnub, post a screenshot of your client panel for tickets, if it isn't there then... I don't know  but maybe SolusVM can provide proof of their side as well...


----------



## MartinD

XFS_Duke said:


> Well myself, I like seeing proof. Anyone can sign up as SVM_Phill and reply to a thread with random numbers... Admins, do yall have proof that this was indeed Phillip Bandelow from SolusLabs?


It is Phill, yes.



netnub said:


> Interesting how it doesn't exist then, eh?


Have you logged in to your account to check? Perhaps even checking your email? Perhaps you can explain why they would delete your ticket... why it makes any kind of sense to ignore the one person who claims to have all this knowledge?


----------



## saliq

Guys whoever has the link please PM me. I really need to see whats in it, I know people are saying we should consider all compressed  but I would like to see for my self.


----------



## XFS_Duke

saliq said:


> Guys whoever has the link please PM me. I really need to see whats in it, I know people are saying we should consider all compressed  but I would like to see for my self.


Not really... If you're a customer... Your information is out there... No need to see others information as well... So stop being stupid... Change your passwords and be done with it...


----------



## Jack

notFound said:


> I have reported the vulns I have been made aware of to SolusVM by that little birdy, it doesn't really take a genuis to figure any of them out once you have access to the un-encoded version of SolusVM. I'm not holding my breath on SolusVM responding.
> 
> (_Yes, and I'm sure I've just given away to everyone who I actually am. ;-)_)


Borat.


----------



## saliq

XFS_Duke said:


> Not really... If you're a customer... Your information is out there... No need to see others information as well... So stop being stupid... Change your passwords and be done with it...


Not stupid. And I dont care about other peoples stuff I want to see my info. If someone can just send me a block with my info. And I did change my passwords right away.


----------



## FHN-Eric

MartinD said:


> It is Phill, yes.
> 
> Have you logged in to your account to check? Perhaps even checking your email? Perhaps you can explain why they would delete your ticket... why it makes any kind of sense to ignore the one person who claims to have all this knowledge?


All this proves is that Curtis did submit a ticket, but was lying about not getting a response. No suprise there that Curtis lied about not getting a response.


----------



## Chronic

Another customer here, VPS located in Buffalo, NY. Dodged the bullet again I guess.


[email protected]:~# uptime
17:24:21 up 30 days, 20:36, 2 users, load average: 0.00, 0.00, 0.00

Messed up stuff. I'm disappointed they didn't send out any emails regarding this - if it weren't for this forum, I would be none the wiser.


----------



## MannDude

FHN-Eric said:


> All this proves is that Curtis did submit a ticket, but was lying about not getting a response. No suprise there that Curtis lied about not getting a response.


Can you start being more specific to which Curtis you're talking about? My name is also Curtis, and I'm not used to having to share my name.

I went to a school where I was the only Curtis, so as you can understand, I don't want their to be any confusion as some people on here know me by 'Curtis' as well.


----------



## XFS_Duke

Chronic said:


> Another customer here, VPS located in Buffalo, NY. Dodged the bullet again I guess.
> 
> 
> [email protected]:~# uptime
> 17:24:21 up 30 days, 20:36, 2 users, load average: 0.00, 0.00, 0.00
> 
> Messed up stuff. I'm disappointed they didn't send out any emails regarding this - if it weren't for this forum, I would be none the wiser.


CVPS did send out an email... Some people didn't get it... Not sure why... but there was an email sent


----------



## FHN-Eric

MannDude said:


> Can you start being more specific to which Curtis you're talking about? My name is also Curtis, and I'm not used to having to share my name.
> 
> I went to a school where I was the only Curtis, so as you can understand, I don't want their to be any confusion as some people on here know me by 'Curtis' as well.


I was talking about Curtis G (netnub)


----------



## SVMPhill

XFS_Duke said:


> Well myself, I like seeing proof. Anyone can sign up as SVM_Phill and reply to a thread with random numbers... Admins, do yall have proof that this was indeed Phillip Bandelow from SolusLabs?


This is to confirm it's me: OQ4W0

You will know what that is.


----------



## AnthonySmith

So what the actual fuck is going on, cvps cried wolf, soluslabs refuse to comment on public facts regarding their own software... I am beyond pissed off.


----------



## XFS_Duke

SVM_Phill said:


> This is to confirm it's me: OQ4W0
> 
> You will know what that is.


Ok, that is proof enough for me


----------



## SVMPhill

Anthony i responded to your support ticket.


----------



## HiveMinded

Another customer here, looks like there was a reboot this morning, I don't see anything wrong so far, but I'm at work and don't have time to look at everything. Last email was a billing statement. 


[email protected] ~]# uptime
12:39:54 up 7:03, 2 users, load average: 0.00, 0.03, 0.00
[email protected] ~]$ last reboot | less
reboot system boot 2.6.32-042stab07 Tue Jun 18 05:36 - 12:36 (07:00)
reboot   system boot  2.6.32-042stab07 Sat May 18 15:36 - 12:47 (30+21:11)

Was wrong they last emailed me about the Redhat vulnerability patch hence the May 18th Reboot


----------



## Lanarchy

My one node that's accessible, but was down at first, was rebooted this morning as well.


----------



## atho

Mine wasnt touched either.

[email protected]*.*:~# uptime
 21:51:44 up 134 days,  1:38,  1 user,  load average: 0.07, 0.02, 0.00


----------



## mystic

Looks like a lot of you got lucky.  Mines been down all day, completely out of commission.  This is directly impacting my business with clients...


----------



## jfreak53

Does anyone have access to the list still?? I need to check for a couple usernames on there, don't really want to change passwords if I don't have to on about 10 of them


----------



## crspyjohn

XFS_Duke said:


> CVPS did send out an email... Some people didn't get it... Not sure why... but there was an email sent


 I'm assuming they're going to notify a small portion of their customer base about the hack. When customers complain they never received an email about the hack, one of those customers would likely speak up and say they did receive the email. They'll chalk it up as the email ending up in the spam box or being denied by your mailing provider, makes it look like they did their due diligence.


----------



## XFS_Duke

jfreak53 said:


> Does anyone have access to the list still?? I need to check for a couple usernames on there, don't really want to change passwords if I don't have to on about 10 of them


Just change your passwords... No need to get the list... Nor is there any real reason to ask for it... Just change your passwords to be safe... Just because it may or may not be in the DB dump doesn't mean that they don't have it themselves... Cover your ass and change your passwords... Make it easy on yourself...


----------



## jacobsta811

I've got 4 nodes on ChicagoVPS, all down, one each in Atlanta, Los Angeles, Chicago, and Buffalo. Buffalo node sent me some emails from a cron job before it went down saying that the drupal directory was gone, so I assume at least the buffalo server was attempted to be deleted. I hope they *don't* put my nodes back up without changing the root password first. Given the speed of port scans, it seems likely that hackers could get to some of my boxes before I can change the password. I am also curious whether the "central backup" backups created from within SolusVM can be restored - you only get one slot per server but I just did that a few days ago and would lose basically zero data or setup time if those can be restored.

I disagree no reason to see the list. If my server root password is in plain text, it means I have to backup data, reinstall and start fresh to be sure I am not compromised. If it isn't or isn't right, I can probably just check the server carefully after changing the password. I always did my VPS by reinstalling and then changing my root password from the one sent initially in SolusVM rather than over SSH, so I expect my passwords are probably in plain text, but I'd still like to know for sure.


----------



## upsetcvps

XFS_Duke said:


> Just change your passwords... No need to get the list... Nor is there any real reason to ask for it... Just change your passwords to be safe... Just because it may or may not be in the DB dump doesn't mean that they don't have it themselves... Cover your ass and change your passwords... Make it easy on yourself...


There is a good reason to ask for it.  Namely, to know what information exactly was compromised.


----------



## netsat

24khost said:


> I am not sure who netstat is but signed his post /johnny which I figured since Eric is here JohnnyDbag can't be far behind.
> 
> notFound not sure haven't read alot of your posts, so can't tell who you are based on this thread.


I don't know who JohnnyDbag is.

I am from Denmark - manndude can verify my ip.

I just use my real name - not an alias like many others her.

/Johnny Andersen


----------



## XFS_Duke

crspyjohn said:


> I'm assuming they're going to notify a small portion of their customer base about the hack. When customers complain they never received an email about the hack, one of those customers would likely speak up and say they did receive the email. They'll chalk it up as the email ending up in the spam box or being denied by your mailing provider, makes it look like they did their due diligence.


True, but a few people said they got the email already and if i'm not mistaken they posted an announcement... Haven't checked their announcement yet though...


----------



## jfreak53

jacobsta811 said:


> I've got 4 nodes on ChicagoVPS, all down, one each in Atlanta, Los Angeles, Chicago, and Buffalo. Buffalo node sent me some emails from a cron job before it went down saying that the drupal directory was gone, so I assume at least the buffalo server was attempted to be deleted. I hope they *don't* put my nodes back up without changing the root password first. Given the speed of port scans, it seems likely that hackers could get to some of my boxes before I can change the password. I am also curious whether the "central backup" backups created from within SolusVM can be restored - you only get one slot per server but I just did that a few days ago and would lose basically zero data or setup time if those can be restored.
> 
> I disagree no reason to see the list. If my server root password is in plain text, it means I have to backup data, reinstall and start fresh to be sure I am not compromised. If it isn't or isn't right, I can probably just check the server carefully after changing the password. I always did my VPS by reinstalling and then changing my root password from the one sent initially in SolusVM rather than over SSH, so I expect my passwords are probably in plain text, but I'd still like to know for sure.





upsetcvps said:


> There is a good reason to ask for it.  Namely, to know what information exactly was compromised.


AGREED! That was my reasoning, 4 of the ten servers are not just web pages but very sensitive data. If they were compromised then I have a LOT of work todo while as if they are not on the list then there is no point in working that much, just changing passwords.


----------



## Chankster

Assume any password you used is compromised and just change your passwords already.


----------



## JDiggity

netsat alright will take your word since dbag signed up!


----------



## XFS_Duke

upsetcvps said:


> There is a good reason to ask for it.  Namely, to know what information exactly was compromised.


Wow, if you've been reading, you know that their entire solusvm database was dumped to the public. Meaning... If you had a VPS with them, your information is *compromised*... If someone with the data wants to pm you that they have it and give you the info that they have on YOU then thats cool, but theres no reason to have the entire database dump... Especially if you have been reading the forums and reading about what has happened...


----------



## drmike

mnsalem said:


> Around 3am Eastern Standat Time (EST) today, there was a security breach, due to a vulnerability in SolusVM that allowed a command line to be run to dump the ChicagoVPS SolusVM client database and attempt to delete all data from our nodes. Our staff is working tirelessly to get everything back online, along working with SolusVM to address the root issue and no furthur impact is expected.


 

3AM eastern?  Wrong.

The hack was like 24 hours prior - when the entire SolusVM customer database was taken.

The physical servers failing and being deleted, 3AM?    It was earlier than that.


----------



## jfreak53

XFS_Duke said:


> Wow, if you've been reading, you know that their entire solusvm database was dumped to the public. Meaning... If you had a VPS with them, your information is *compromised*... If someone with the data wants to pm you that they have it and give you the info that they have on YOU then thats cool, but theres no reason to have the entire database dump... Especially if you have been reading the forums and reading about what has happened...


At no point did I ever "ASK" for the entire dump, I asked if someone had it. In that case if they do I can PM them the users and ask them nicely if they can grep the file to see if my 10 users are there. Simple.

Again, this is NOT about changing passwords, I already did that. It is about the data within the system. Out of the 10, 5 are back online, meaning if they were not brought back from a backup from cVPS the data contained could be compromised (messed with!!), meaning I have more work to do than just changing a password.

This is the reason I am curious, not changing passwords.


----------



## chronos511

This is an example of the info that was leaked in the db:

cvps_???? (assigned by CVPS) (hash of password)=(some have this, some don't. Original password used to sign up) [email protected] first lastname

cvps_???? (same as above) (IP of server) (name of server) (hypervisor) (OS and version) (RAM) (OG password as above)


----------



## Mun

jfreak53 said:


> At no point did I ever "ASK" for the entire dump, I asked if someone had it. In that case if they do I can PM them the users and ask them nicely if they can grep the file to see if my 10 users are there. Simple.
> 
> Again, this is NOT about changing passwords, I already did that. It is about the data within the system. Out of the 10, 5 are back online, meaning if they were not brought back from a backup from cVPS the data contained could be compromised (messed with!!), meaning I have more work to do than just changing a password.
> 
> This is the reason I am curious, not changing passwords.



This has happened before with CVPS, so simply put, your old passwords are now floating around the internet on 50 mirrors with IP addresses to try your password on.

Mun


----------



## jacobsta811

The server that started emailing me, presumably in mid deletion, happened at 3:25AM EDT. DB could have been hacked well before that though, and possibly some targeted attacks performed before the dump ever got posted.


----------



## upsetcvps

I sent you a pm, jfreak53


----------



## upsetcvps

jacobsta811 said:


> The server that started emailing me, presumably in mid deletion, happened at 3:25AM EDT. DB could have been hacked well before that though, and possibly some targeted attacks performed before the dump ever got posted.


3:15AM EDT for me I lost connection


----------



## drmike

If anyone still wants to know if their email appears in the dump, PM me and I'll run a search for you.

The dump doesn't include all accounts/some prior customer accounts.


----------



## MartinD

netnub said:


> Interesting how it doesn't exist then, eh?


Still waiting..... you're lurking but posting nada.


----------



## Lanarchy

My NY node is responding to ping, but inaccessible. The count so far

1 fully functional

2 responding to ping but inaccessible

2 fully down


----------



## insaneguy

One of my VPS's is still down the other went down for a while and came back luckily that is the one with my most important customers data.


----------



## Lanarchy

My node that just came up has a new kernel.

Linux 2.6.32-042stab078.22

And another says

Linux 2.6.32-042stab076.8

Could have been my doing and it just rebooted for me to see it. But that's different from what I remember.

On both, yum says up to date.


----------



## nunim

Which SolusVM file was exploited this time?


----------



## MartinD

None - it's nonsense.


----------



## jfreak53

MartinD said:


> None - it's nonsense.


Really, what your theory? ha ha


----------



## nunim

MartinD said:


> None - it's nonsense.


Well, ChicagoVPS could just be incompetent and been rooted from the centralbackup exploit, but netn00b posted some "code"  that he claimed was responsible, was looking for what file that was from.

Isn't CurtisG (netnub) the guy who was selling "dedicated servers" that were just shell accounts?


----------



## jfreak53

Considering cVPS is not the only one effected and Solus has launched their own post on the subject on their site, I'd fair to say it's the exploit


----------



## MartinD

Yes, that's the same person.

...how much weight do you want to put on what he says? Also, the code he showed wasn't a vulnerability either. He's obviously decided any instance of 'exec' in any kind of php code is a vulnerability.


----------



## MartinD

jfreak53 said:


> Considering cVPS is not the only one effected and Solus has launched their own post on the subject on their site, I'd fair to say it's the exploit


That was a different exploit that was patched. Solus held up their hands to that, too.


----------



## nunim

MartinD said:


> Yes, that's the same person.
> 
> ...how much weight do you want to put on what he says? Also, the code he showed wasn't a vulnerability either. He's obviously decided any instance of 'exec' in any kind of php code is a vulnerability.


Which is why I'm trying to figure out what he claims the exploit to be so I can looksie,  it's fairly trivial to decode ioncube, not that I would do such a thing...

I would also take whatever CVPS Chris says with a grain of salt, as he couldn't explain their last hack that had their db released...  Good thing they have backups this time, except in Atlanta it seems?


----------



## chronos511

Anyone else suddenly unable to get LET to load?


----------



## mmance

Chris has been very vague in his response to me personally today.  







I also had someone grep my username for the Client Area in the stolen data.  It came back 0 results.


----------



## drmike

Ran a bunch of lookups for folks here to see if their details were in the dump.

I can confirm if you cancelled your services after the last hack in November - February, your details probably aren't in there.

Anyone else want info looked up, PM me.  

Will be back in a bit.


----------



## mnsalem

Just thought to drop by and mention that i just got the email with the report (that update which was posted several hours ago).


----------



## DaringHost

chronos511 said:


> Anyone else suddenly unable to get LET to load?


See: http://vpsboard.com/topic/770-lowendtalkcom-down/


----------



## mmance

Chris has been very vague in his response to me personally today.  






I also had someone grep my username for the Client Area in the stolen data.  It came back 0 results.


----------



## saliq

mmance said:


> Chris has been very vague in his response to me personally today.
> 
> 
> 
> 
> 
> 
> I also had someone grep my username for the Client Area in the stolen data.  It came back 0 results.


If your site and email is same as the username here then you are in it


----------



## upsetcvps

mmance said:


> Chris has been very vague in his response to me personally today.
> 
> 
> 
> 
> 
> 
> 
> I also had someone grep my username for the Client Area in the stolen data.  It came back 0 results.


yes, your e-mail address would not be hard to guess based on your username, Marc


----------



## jfreak53

Offline/Online Nodes:

http://stats.pingdom.com/jzrszp4wfu79


----------



## drmike

jfreak53 said:


> Offline/Online Nodes:
> 
> http://stats.pingdom.com/jzrszp4wfu79



From Pingdom's monitoring shows 19 servers that are broken....


----------



## mnsalem

they are indeed working on it! Last time i checked pingdom, 3 out of the 4 servers in atlanta were offline! now Just 1 is left.

Same for Buffalo! 4 servers were down .. now just 2 ... and i happen to be on the one that is down (facepalm)


----------



## upsetcvps

mnsalem said:


> they are indeed working on it! Last time i checked pingdom, 3 out of the 4 servers in atlanta were offline! now Just 1 is left.
> 
> 
> Same for Buffalo! 4 servers were down .. now just 2 ... and i happen to be on the one that is down (facepalm)


We are probably in the same server.  How do you know what server you are on?


----------



## mnsalem

upsetcvps said:


> We are probably in the same server.  How do you know what server you are on?


I am on 192.227.129.xxx subnet ... that's BUF19. through the CP back in its working days.


Anything in buffalo other than that will be on BUF17


----------



## TheLinuxBug

I think this thread should just be closed.  If there is anymore real news about this, I think we can open a new thread, or even better, post it in the cest pit.  There is enough CVPS PR threads open here already.

Cheers!


----------



## HalfEatenPie

This is just ridiculous.  Closed.


----------



## HalfEatenPie

Ok this topic has been re-opened after cleaning up a bit.  Please keep the discussion focused on the topic.  The other discussion can be found here: http://vpsboard.com/topic/777-personal-arguments/


----------



## jfreak53

Thanks mod for cleaning this mess up.

You know cVPS an update no matter how small it is would really be helpful, even if it is small.


----------



## CVPS_Chris

Jfreak, we are still working to get the remaining nodes online.


----------



## Marc M.

How much warning do you need as a software provider about your code being poorly written? And why do you write code like this? Sorry, but I can't fault any provider that was hit by this attack, and all I can say is that I am sorry that some of you guys had to suffer because of this:


<?php
if ($_POST['delete']) {
$xc = $db -> query('SELECT * FROM centralbackup WHERE id = \'' . $_POST['deleteid'] . '\'', true);
#[...]
if ($xc[status] == 'failed') {
exec('php /usr/local/solusvm/system/bus.php -- --comm=deletebackup --serverid=' . $xc['bserver'] . ' --nodeid=' . $vdata['nodeid'] . ' --vserverid=' . $vdata['vserverid'] . ' --filename=' . $xc['filename']);
#[...]
}
}
?>


D_Strout said:


> Hasn't anyone decrypted the source? Couldn't they then run a search for dumb execs?


*@D. Strout* There's been a decoded version floating around the web for a while now, I guess that's how the vulnerability was found and exploited in the first place. Pretty lame, but it is what it is.

Guys, here is something simple that you should do immediately: restrict access to the admin path. Restrict it by IP, with a password, or ideally both. @Kujoe had some good advice as well on how to secure SolusVM.

Kind regards,

Marc


----------



## concerto49

Has anyone heard back from Solus yet?


----------



## MannDude

concerto49 said:


> Has anyone heard back from Solus yet?


I wouldimagine they're quite busy attempting damage control.


----------



## Mun

MannDude, I know you don't work there, but Urpad got hit too?


----------



## Otakumatic

Mun said:


> MannDude, I know you don't work there, but Urpad got hit too?


Their site works for me.


----------



## upsetcvps

marcm said:


> How much warning do you need as a software provider about your code being poorly written? And why do you write code like this? Sorry, but I can't fault any provider that was hit by this attack, and all I can say is that I am sorry that some of you guys had to suffer because of this:
> 
> 
> <?php
> if ($_POST['delete']) {
> $xc = $db -> query('SELECT * FROM centralbackup WHERE id = \'' . $_POST['deleteid'] . '\'', true);
> #[...]
> if ($xc[status] == 'failed') {
> exec('php /usr/local/solusvm/system/bus.php -- --comm=deletebackup --serverid=' . $xc['bserver'] . ' --nodeid=' . $vdata['nodeid'] . ' --vserverid=' . $vdata['vserverid'] . ' --filename=' . $xc['filename']);
> #[...]
> }
> }
> ?>
> *D. Strout* There's been a decoded version floating around the web for a while now, I guess that's how the vulnerability was found and exploited in the first place. Pretty lame, but it is what it is.
> 
> Guys, here is something simple that you should do immediately: restrict access to the admin path. Restrict it by IP, with a password, or ideally both. @Kujoe had some good advice as well on how to secure SolusVM.
> 
> Kind regards,
> 
> Marc


what. the. fuck.


----------



## MannDude

Mun said:


> MannDude, I know you don't work there, but Urpad got hit too?


Yeah, don't work there anymore.

I messaged Jason earlier this morning and told him what was going on and it may be best to shut the Solus master off for a while.

Doesn't matter, Adam Ng ("Kevin Hillstrand") has had the URPad WHMCS and SolusVM DB (both dated) for a while and has always threatened to post it anytime we made him mad. I'd change your passwords anyways since I could never get the old owner to force password resets on everyone, nor have the new owners yet. Both parties have indeed been informed that this kid has dated DBs and has threatened, multiple times, to post them if we don't back off on things that upset him. (Like poking the Adam/Kevin thing, etc)


----------



## Amitz

That's somehow unrelated, but is this 'Adam Ng' in any way related to Adam, the former owner of VPSLatch? I still have a bone to pick with that a**hole...


----------



## drmike

MannDude said:


> Adam Ng ("Kevin Hillstrand") has had the URPad WHMCS and SolusVM DB (both dated) for a while and has always threatened to post it anytime we made him mad


What the f*Ck!?!?!?!

Where is @Miller?


----------



## MannDude

Amitz said:


> That's somehow unrelated, but is this 'Adam Ng' in any way related to Adam, the former owner of VPSLatch? I still have a bone to pick with that a**hole...


Yes. That requires a thread of it's own, however. Be my guest.


----------



## drmike

MannDude said:


> Yes. That requires a thread of it's own, however. Be my guest.



Gladly, posting a new thread now.


----------



## concerto49

MannDude said:


> Yes. That requires a thread of it's own, however. Be my guest.


Liam @ LET patched this 0-day exploit yesterday. CVPS_Kevin got renamed to CVPS_Adam.


----------



## netnub

So wait, I'm not allow to post code snippits, but he IS?


----------



## Francisco

netnub said:


> So wait, I'm not allow to post code snippits, but he IS?


The snippet from above was the source of the last exploit. If there's new code and solus patches it? You're then "fine" to post it since you've at least done due diligence by the vendor.

0-day'ing it is seen as 'poor taste' 

Francisco


----------



## Mun

netnub said:


> So wait, I'm not allow to post code snippits, but he IS?


So you are allowed to steal WHMCS, but someone else isn't?

So you are allowed to scam people, but someone else isn't?

So you are allowed to steal databases, but someone else isn't?

These are all related to you, and it is getting to the point that you really need to grow a brain, as well as mature into something more then a sniveling rat.


----------



## Dan

MannDude said:


> I messaged Jason earlier this morning and told him what was going on and it may be best to shut the Solus master off for a while.


 

Would of been nice of them to contact their clients about this ... Urpads support has started to go down hill too...


----------



## Otakumatic

Didn't they sell URPad a while back? I thought I read about a bunch of changes at URPad on LET a while back....


----------



## MannDude

athk said:


> Would of been nice of them to contact their clients about this ... Urpads support has started to go down hill too...


I assumed they would have. Out of my hands.



Otakumatic said:


> Didn't they sell URPad a while back? I thought I read about a bunch of changes at URPad on LET a while back....


Yes, towards the beginning of May. First or second week. Can't remember.


----------



## fileMEDIA

Solusvm 1.14.00 BETA R5 is available..no changelog yet.

This is an important security fix. You are encouraged to update as soon as possible. A full detailed report will be published at a later date.


----------



## Mun

fileMEDIA said:


> Solusvm 1.14.00 BETA R5 is available..no changelog yet.


Changelog:

Removed old exploits that we forgot about

Added new exploits so we can see how well our panel is doing

Added a new feature to DDOS Stallion cause it is too good.

Created a function to ask for confirmation if you want to delete all nodes, just to make sure the hacker really wants too.

Added a Clarke button that pops up a picture of him.

Created a new function so rofl.php show a picture of a dog when it is used against the newer version because we don't like that guy.

Added new feature to make it look like CVPS is incompetent, though we really didn't need to do much.

Claimed everything is Green now, since we use more code, that does less.

Called up our lawyers to see if we are going to get sued, and they told us nope as long as you add this little tid bit in the agreement for installing this new patch.

This is all joking of course, or is it O_O

Mun


----------



## john

Looks like there actually was another exploit.


----------



## drmike

^ Mun = classic gold! 

Keep it up.  I needed a laugh.


----------



## weservit

*PLEASE READ THIS INFORMATION CAREFULLY. THIS INFORMATION IS RELEVANT TO ALL VERSIONS OF SOLUSVM, INCLUDING BETA VERSIONS.*

As you may be aware we are currently running a full in house and external code audit. This release contains several important security fixes for all versions of SolusVM.

We highly suggest you update your system as soon as possible. Updates are available through the normal channels.

*Latest Stable Version:* 1.14.00 R5
*Latest Beta Version:* 1.13.05

Please be aware the audit is still underway and more updates may follow.

Thank you for your co-operation and understanding.

Regards,
Soluslabs Security Team


----------



## drmike

Official thing there @weservit?  Have a URL to confirm that?

Glad to see Soluslabs getting off their arses and doing something other than denying.


----------



## Marc M.

*@**Mun*,

I would add this: "Created a function to delete all VMs from all nodes just to see if the hacker can find it..." ROFL

and

this: "Added dead simple functionality to facilitate a MySQL injection exploit to see if anyone could find it in less than two months..." again ROFL ... well, it took the hacker almost two months minus four days to find it. IIRC about two months ago someone posted a iDezender decoded SolusVM online.

Classic Gold Mun :lol:


----------



## MannDude

buffalooed said:


> Official thing there @weservit?  Have a URL to confirm that?
> 
> Glad to see Soluslabs getting off their arses and doing something other than denying.


http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/


----------



## weservit

http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/

Also received an email from them now.


----------



## mikho

Looks like they are doing a better job then zamfoo.


----------



## Marc M.

Looks official to me: http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/


----------



## weservit

I see multiple modified files in the /usr/local/solusvm/www folder. Looks like they found more than 1 exploit..


----------



## Marc M.

weservit said:


> I see a lot of modified files in the /usr/local/solusvm/www folder. Looks like they found more than 1 exploit..


*@**weservit* I'm glad that something got them from sitting around on their d**** all day long and finally doing a full security audit. This begs the questions if a disaster is necessary every time for them to do something about it?!


----------



## concerto49

weservit said:


> I see multiple modified files in the /usr/local/solusvm/www folder. Looks like they found more than 1 exploit..


Of course, at least 3 were reported directly to them as of yesterday.


----------



## Mun

weservit said:


> I see multiple modified files in the /usr/local/solusvm/www folder. Looks like they found more than 1 exploit..



Shhh, they really added new ones.

Mun


----------



## Marc M.

Mun said:


> Shhh, they really added new ones.


*@**Mun* it's either that or they are paying their coders so poorly that every so often their employees plant one or two Easter eggs in the code. Since they don't audit it unless a disaster like this one happens, no one cares. I don't see them jumping on their swords any time soon because they've messed up.


----------



## Mun

marcm said:


> *@Mun* it's either that or they are paying their coders so poorly that every so often their employees plant one or two Easter eggs in the code. Since they don't audit it unless a disaster like this one happens, no one cares. I don't see them jumping on their swords any time soon because they've messed up.


Or they had so much bad press that an addition to a line here and there makes it all better.

Mun


----------



## ItsGermy

CVPS_Chris said:


> Jfreak, we are still working to get the remaining nodes online.


This isn't helpful. We've been down for almost 24 hours now and some sort of regular updates as to where you're at with restores and an ETA for the remaining nodes would be great.

Please don't hide behind the typical excuses of, "We don't have time to update...., We're dedicating all our resources....". Your customers need information and they need better information than, "We're working on it...."


----------



## Aldryic C'boas

ItsGermy said:


> We don't have time to update....


Well, he did find the time to come in here and try to brush off the Adam/Kevin situation, so I'm sure he'll at least make just as much time to post more status updates ASAP. To do otherwise would just be downright insulting to the clients waiting to hear something important.


----------



## maounique

The question is:

Is it safe to put it back on ?

I would say they patched so far the exploits that have been shown to them.

There should be others because I dont buy that audit stuff they are claiming.

Basically it is like this:

1. Solus hack on CVPS. Solus says they did an audit and it is not their fault;

2. Centralbackup disaster strikes. Solus can no longer say there is no exploit, it takes them HOURS, at least half a day after the disclosure to release a fix, but they do aknowledge it;

3. CVPS hacked again, Solus again sais it wasnt their fault, they claim there is no exploit, they were not notified, etc, the classical dance;

4. They release a fix after an "audit" saying there are more to come.

If there was no 4, I am sure some folks started to believe them there is no exploit and CVPS and others are lying, as I started to think maybe it is the time to bring solus back online.

In the light of these events, we are considering bringing solus back but allow only the IPs of salvatore and me to access it, as well as the billing panels.

This is beyond ridiculous, what a bunch of clowns...


----------



## concerto49

Mao said:


> The question is:
> 
> Is it safe to put it back on ?
> 
> I would say they patched so far the exploits that have been shown to them.
> 
> There should be others because I dont buy that audit stuff they are claiming.
> 
> Basically it is like this:
> 
> 1. Solus hack on CVPS. Solus says they did an audit and it is not their fault;
> 
> 2. Centralbackup disaster strikes. Solus can no longer say there is no exploit, it takes them HOURS, at least half a day after the disclosure to release a fix, but they do aknowledge it;
> 
> 3. CVPS hacked again, Solus again sais it wasnt their fault, they claim there is no exploit, they were not notified, etc, the classical dance;
> 
> 4. They release a fix after an "audit" saying there are more to come.
> 
> If there was no 4, I am sure some folks started to believe them there is no exploit and CVPS and others are lying, as I started to think maybe it is the time to bring solus back online.
> 
> In the light of these events, we are considering bringing solus back but allow only the IPs of salvatore and me to access it, as well as the billing panels.
> 
> This is beyond ridiculous, what a bunch of clowns...


1. The first hack no one has published evidence on what happened.

2. That was explicit and acknowledged by Solus.

3. Solus didn't say it wasn't their fault in this 2nd hack this time around.

4. More like we and others reported the exploits.


----------



## Mun

ItsGermy said:


> This isn't helpful. We've been down for almost 24 hours now and some sort of regular updates as to where you're at with restores and an ETA for the remaining nodes would be great.
> 
> Please don't hide behind the typical excuses of, "We don't have time to update...., We're dedicating all our resources....". Your customers need information and they need better information than, "We're working on it...."



Then find a new host. You are asking way too much from Cvps_chris, and I have told him this before. You bought a service with a company with a rep. for not giving out informative updates. 

Here is a list of some other providers: http://vpswiki.us/


----------



## maounique

concerto49 said:


> 1. The first hack no one has published evidence on what happened.
> 
> 2. That was explicit and acknowledged by Solus.
> 
> 3. Solus didn't say it wasn't their fault in this 2nd hack this time around.
> 
> 4. More like we and others reported the exploits.


1. In light of what happened later, does anyone need any evidence ?

2. Yeah, I wonder if it was not disclosed so brutally, would it have been the same ?

3. They did, kept saying like the first CVPS hack that there is no evidence, blah-blah.

4. Yes, the audit is a another hoax like the previous audit that yielded no proof there is an explot to be blamed for cvps hack. They seem to slowly aknowledge and patch only the publicly disclosed holes, therefore, instead of condemning, I commendd the people that did this.

The way solus handled it so far makes me believe the poeple claiming their private reports were ignored. In light of latest events, soluls looks THAT bad.


----------



## concerto49

Mao said:


> 1. In light of what happened later, does anyone need any evidence ?
> 
> 2. Yeah, I wonder if it was not disclosed so brutally, would it have been the same ?
> 
> 3. They did, kept saying like the first CVPS hack that there is no evidence, blah-blah.
> 
> 4. Yes, the audit is a another hoax like the previous audit that yielded no proof there is an explot to be blamed for cvps hack. They seem to slowly aknowledge and patch only the publicly disclosed holes, therefore, instead of condemning, I commendd the people that did this.
> 
> The way solus handled it so far makes me believe the poeple claiming their private reports were ignored. In light of latest events, soluls looks THAT bad.


What's your take then? Let's collaborate and build a new panel shall we?  :lol:


----------



## netnub

And you guys stated I was kidding about vulnerabilities. http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/


----------



## maounique

concerto49 said:


> What's your take then? Let's collaborate and build a new panel shall we?  :lol:


I believe joepie91 was already on something like that ?

We would gladly donate something to the project as long as it remains open source.

I lost hope solus would wake up after this (yet another) disaster, their whole preocupation now looks like leaning towards damage control, how much can be still denied and how much they have to aknowledge. That is no way to act in this business.


----------



## SVMPhill

concerto49 said:


> Of course, at least 3 were reported directly to them as of yesterday.


Do you have more information on this please. No reports were made.


----------



## Mun

netnub said:


> And you guys stated I was kidding about vulnerabilities. http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/



and you stated you changed a leaf and wouldn't do anything "fishy" any longer, yet you still do. 

Mun


----------



## drmike

@netnub,  I don't doubt you.  Lots of folks are in cover-their-ass mode and protect-their-friends mode.

Does SolusVM know about the other exploits now?  Have they responded to you?


----------



## netnub

SVM_Phill said:


> Do you have more information on this please. No reports were made.


Why don't you ask good old Humza who I gave him snippits to give to you.

Don't act dumb, it really bugs me.


----------



## concerto49

SVM_Phill said:


> Do you have more information on this please. No reports were made.


Yes they were. Infinity reported it. Raised a ticket. A lot of others followed. Go through the tickets escalated.


----------



## drmike

@netnub, PM me the info and I'll PERSONALLY make sure Phil gets it and anyone else you think should and I'll verify everyone has received it.

Tired of the run around/miscommunications/posts being pulled --- going on around the exploits.


----------



## netnub

Mun said:


> and you stated you changed a leaf and wouldn't do anything "fishy" any longer, yet you still do.
> 
> Mun


Fishy? More like helping security.


----------



## Mun

netnub said:


> Fishy? More like helping security.


Security, more like stealing and lying to your customers. 

What I am saying, is you have done things in the past that are ironic in this case. You really need to clean up your image.

Mun


----------



## netnub

My image is clean, you're bringing up the past, so how about I start bringing up the past? Like the past where I disclosed vulnerabilities to the public.

Shall I make that the future also?


----------



## drmike

Mun said:


> you have done things in the past that are ironic in this case.


 

Shit dawg, that description applies to so many in this industry.  Whole bunch of folks just got a strange facial tick and did a double take on that while skimming


----------



## Mun

netnub said:


> My image is clean, you're bringing up the past, so how about I start bringing up the past? Like the past where I disclosed vulnerabilities to the public.
> 
> Shall I make that the future also?


I'm going to stop derailing this thread, but my point is that you have done some shitty things.

Mun


----------



## PcJamesy

Wow this new post just came out. The last threat was to about SolusVM exploit. This one can't be good.


----------



## Magiobiwan

And then the past where you threatened to hack, DDoS, and exploit several hosts for no apparent reason. And then claimed it was someone ELSE behind all this.


----------



## PcJamesy

Looks like were going to have to pull both panels down soon, who will fall victim this time.


----------



## drmike

@PcJamesy, where did that thread copied from?


----------



## PcJamesy

buffalooed said:


> @PcJamesy, where did that thread copied from?


http://www.lowendbox.com/blog/a-days-recap-solusvm-exploit-released-ramnode-downtime-and-robert-clarke/#comment-121284


----------



## netnub

buffalooed said:


> @PcJamesy, where did that thread copied from?


http://www.lowendbox.com/blog/a-days-recap-solusvm-exploit-released-ramnode-downtime-and-robert-clarke/#comment-121284


----------



## MannDude

WHMCS news requires it's own thread...

The hosting industry should be on high alert it appears.


----------



## drmike

Start a new thread for the WHMCS stuff @MannDude...


----------



## Marc M.

*@* at the bottom http://www.lowendbox.com/blog/a-days-recap-solusvm-exploit-released-ramnode-downtime-and-robert-clarke/


----------



## johnnyd95

&nbsp;



PcJamesy said:


> Wow this new post just came out. The last threat was to about SolusVM exploit. This one can't be good.
> 
> 
> &nbsp;


&nbsp;

Me and Curtis G are releasing 0 day vun for whmcs friday


----------



## maounique

Actually, I know how insecure we are, problem is, what to do ?

Leave only linode and amazon provide VPSes ? Will you feel secure with the gov't having a direct line into your stuff ?

Not that they are bullet-proof, anyway, I wish ppl will focus more on taking down the establishment than the little folks with a small business.


----------



## drmike

True 'dat Chairman Mao!

Government isn't spying, they just are providing free backups for your data to every citizen   You just didn't get your API access key yet.


----------



## Otakumatic

There is no way I'm paying $20 for a Linode VPS when I can get the same specs for ~$7.

And not all LEB VPSers are script kiddies. :|


----------



## Marc M.

Otakumatic said:


> There is no way I'm paying $20 for a Linode VPS when I can get the same specs for ~$7.


*@**Otakumatic* Actually you will get better performance for your $7. Heck, our Xen nodes provide way better I/O than Linode, and we're releasing our SSD Cached Xen packages soon (very soon). And there are many other small providers who offer quality service for $7. So of course Linode is making a killing by pushing as many users as possible per node. Plus I have a sneaky suspicion that they are using RAID 5 or 6 instead of 10 to get more storage space out of their drives.


----------



## yolo

johnnyd95 said:


> &nbsp; &nbsp;
> 
> 
> Me and Curtis G are releasing 0 day vun for whmcs friday


*Curtis G and I


----------



## drmike

Spencer said:


> *Curtis G and I


 

Dude they are hackers.  They can hack Engwish too.


----------



## texteditor

buffalooed said:


> Dude they are hackers.  They can hack Engwish too.


Grammar is for the sheeple in meatspace


----------



## Magiobiwan

Seriously you two. What are you getting out of this? Lulz? It's not helping the community any. Providers are locking down their stuff, unwilling to risk being compromised, which inconveniences their clients. The node wiping is causing people to lose their data, their time, the effort they've put in to setting stuff up, in some cases money and their own clients, and possibly their livelihoods. If you want to HELP the community (foreign idea, I know), let SolusVM and WHMCS know of the exploits BEFORE releasing them. Once you've informed the companies about the exploits and they've had a reasonable amount of time to respond, THEN you can release the code. Back to what YOU'RE getting out of this. Nothing really. Public hatred towards you. Potential legal action taken against you (civil and/or criminal), with potential jail time and/or monetary fines. You're ruining your future with this. So STOP. I'm sure the rest of the community agrees on this point. It's not helping ANYONE, just hurting. So don't do it any longer.


----------



## maounique

Magiobiwan said:


> let SolusVM and WHMCS know of the exploits BEFORE releasing them. Once you've informed the companies about the exploits and they've had a reasonable amount of time to respond, THEN you can release the code.


I think they did that ?

However, those companies are more interested by PR and spinning the things around instead of the quality of the code.

We have plenty of evidence about that, at least from Solus, I tend to believe them when they say they sent the exploits not only to the companies, but also to infinity and others.

From where I stand, they are doing a good thing, destroying company credibility means they will have to get it back by releasing a decent product for a change.

Everyone will benefit in the end, exploits will no longer stay hidden to be used only by criminals, the fixes will be forced out of the culprits, people will be more aware of security and will take more back-ups as well as not disclosing personal data, everyone will win.

Even solus will have a better product which will generate better sales if they are really thinking about changing their ways, fire a few PR spin doctors and hire better coders. After all, they are not a political party, just a company which has to deliver a product.


----------



## drmike

Mao said:


> fire a few PR spin doctors and hire better coders. After all, they are not a political party, just a company which has to deliver a product


Chairman Mao is on fire!  So true.  Better coders and less PR spin.


----------



## peterw

What a show!

First SolusVM and now Hostbill and WHMCS. It's exciting to see how vulnerable a monoculture is.


----------



## Marc M.

peterw said:


> What a show! First SolusVM and now Hostbill and WHMCS. It's exciting to see how vulnerable a monoculture is.


*@**peterw* yeah, I imagine it is. Imagine how exciting it will be when you will have to pay $30 for the lowest end VPS and close to a $100 to get something decent, like it was just a few short years ago. Then you'll miss this "crappy monoculture" that you like so much to laugh at!


----------



## travmed

Just got this email update from ChicagoVPS. My question is don't we need access to the SolusVM to initiate a reimage of our server is everything is lost?



> [SIZE=small]This is a further status update to the recent security breach that ChicagoVPS has experienced. We have successfully restored some nodes, and the vast majority of our VPS customers are online, however we have a small percentage of nodes which still need to be worked on. Some of the nodes we are working on had data loss that we cannot restore. These nodes are LA18, ATL1, ATL4, ATL5. I you on are on one of these nodes you can safely start to rebuild, or open a ticket asking for this months refund.
> 
> On a positive note, it seems like SolusVM has released a new update in light of the recent incidents ( [/SIZE][SIZE=small]http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/[/SIZE][SIZE=small] ). However, at this time we do not feel comfortable enabling SolusVM access at this minute as we were a victim of their security vulnerabilities two times in the past 7 months. We are evaluating other alternative panels, but at the moment our priority is taking care of our customers and getting the impacted nodes back in working order.
> 
> Please understand that we have all hands on deck working tirelessly to restore service connectivity for those impacted. Therefore, our ticket response times are affected to allow us to effectively work without anything slowing us down. We apologize for the delayed ticket response times but we assure you we are making progress and working hard to get everything back to normal.
> 
> Our goal is to have everything 100% restored tomorrow. Those affected by this incident will recieve compensation.
> 
> Regards,
> 
> ChicagoVPS Team[/SIZE]


----------



## peterw

marcm said:


> *@peterw* yeah, I imagine it is. Imagine how exciting it will be when you will have to pay $30 for the lowest end VPS and close to a $100 to get something decent, like it was just a few short years ago. Then you'll miss this "crappy monoculture" that you like so much to laugh at!


I am pissed off. Someone is trying to destroy the whole SolusVM based economy. I am seeing it but I can't believe it. If the Hostbill and WHMCS 0day exploits are true it is just the beginning.

How should providers work if they can't use SolusVM and WHMCS?

I am using this monoculture too. Nothing to laugh at!


----------



## Marc M.

peterw said:


> I am using this monoculture too. Nothing to laugh at!


*@**peterw* As long as providers take steps to secure them, they will be fine. There are plenty of simple solutions to prevent SQL injections and such, and on top of that providers can use CloudFlare as a reverse proxy (it's running Nginx as well). So no, the entire industry won't come crashing down.


----------



## drmike

travmed said:


> Just got this email update from ChicagoVPS. My question is don't we need access to the SolusVM to initiate a reimage of our server is everything is lost?


I won't ask where and node you are on.  But like the last hack and fail at CVPS, they lost customer VPSes. 

. These nodes are LA18, ATL1, ATL4, ATL5.

As an end user you likely have clue which server you are on or want to waste an hour trying to figure that out.

I'd send them a ticket and ask if they lost your VPS or not.


----------



## zero

My Customers threat with lawsuit. 

ChicagoVPS cant answer the tickets.

When system up and running ?

I need net time for system up and running!


----------



## MannDude

zero said:


> My Customers threat with lawsuit.
> 
> ChicagoVPS cant answer the tickets.
> 
> When system up and running ?
> 
> I need net time for system up and running!


How long have you been down for?

I'd imagine they're still quite busy and they're likely working on getting everyone back up.

What node or location were you in, out of curiosity?


----------



## zero

27 hr ago system shutdowned

I have 4 VPS

1) Atlanta Location (Important Data)

2) Chicago1 (Important Data)

3) Chicago2 (Low Important)

4) LosAngeles (Low Important)


----------



## mnsalem

zero said:


> 30 hr ago system shutdowned
> 
> I have 4 VPS
> 
> 1) Atlanta Location (Important Data)
> 
> 2) Chicago1 (Important Data)
> 
> 3) Chicago2 (Low Important)
> 
> 4) LosAngeles (Low Important)


From their reports, the lost data on [SIZE=small]LA18, ATL1, ATL4 and ATL5[/SIZE] is gone ... irrecoverable. If your No. 1 and No. 4 VPS are on any of these .. they're a goner.

But the chicago locations weren't mentioned, so i'm guessing the backup exists for them.

Mine is still down at the moment as well ... BUF19


----------



## Amitz

I still do not understand why people who have "important" data on a VPS do not keep own backups. Really. I even have backups of the most unimportant data. If one of my VPS providers goes down, it will take me a max. of 5 hours to be fully operational at another place. And I am just a "hobbyist"... Shame on all "professionals" for not having backups.


----------



## peterw

zero said:


> My Customers threat with lawsuit.
> 
> ChicagoVPS cant answer the tickets.
> 
> When system up and running ?
> 
> I need net time for system up and running!


Your customers do not have backups? You do not have backups of services you offer?

My estimation for cvps "next week". Go to another provider, untar your backups and point the domains to the new ip. 4 hours of work and everthing is fine.


----------



## zero

I have some backups not at all.

I'm still waiting but CVPS not respond any ticket or not make statement for us "customers"

Whats happen right now? I'm in darkness.


----------



## AnthonySmith

zero said:


> My Customers threat with lawsuit.


No they did not and if they did then you have made promises you cannot keep that is your fault and your responsibility alone, you are no more important than any other customer of CVPS, it will be done when it is done, I have no doubt they are working hard to bring things up opening tickets and updating forums is only distracting them.


----------



## texteditor

zero said:


> 30 hr ago system shutdowned
> 
> I have 4 VPS
> 
> 1) Atlanta Location (Important Data)
> 
> 2) Chicago1 (Important Data)
> 
> 3) Chicago2 (Low Important)
> 
> 4) LosAngeles (Low Important)



You were hosting it on ChicagoVPS, it couldn't have been _that_ important


----------



## zero

AnthonySmith said:


> No they did not and if they did then you have made promises you cannot keep that is your fault and your responsibility alone, you are no more important than any other customer of CVPS, it will be done when it is done, I have no doubt they are working hard to bring things up opening tickets and updating forums is only distracting them.



yes my fault i miss my backups but, i pay for money for service and stability is this cpvs problem not mine. But problem or hack or whatever happed. I 'm wait statement or any respose from cpvs u understand me.


----------



## zero

texteditor said:


> You were hosting it on ChicagoVPS, it couldn't have been _that_ important


ok before sale cvps say to customer/s if your data not important i will host you. had to say If your important data please leave us.

This problem for me, no money no time


----------



## MannDude

zero said:


> yes my fault i miss my backups but, i pay for money for service and stability is this cpvs problem not mine. But problem or hack or whatever happed. I 'm wait statement or any respose from cpvs u understand me.


I think the best case scenario is you may get a free month of service for the servers impacted. I assume they're quite busy, I'm not quite sure what all was impacted nor how many nodes or backups they've got to restore but I'd imagine you're in queue and will get processed soon. I'd refrain from bumping your ticket, as most providers will process requests at the top of the queue who have waited the longest. Responding to your own ticket updates it, and places it back at the bottom of the queue.

Why would your customers sue you? Your SLA should not promise anything more than the SLA CVPS has for you.

Best of luck in getting it all sorted,


----------



## redjersey

peterw said:


> I am pissed off. Someone is trying to destroy the whole SolusVM based economy. I am seeing it but I can't believe it. If the Hostbill and WHMCS 0day exploits are true it is just the beginning.
> 
> How should providers work if they can't use SolusVM and WHMCS?
> 
> I am using this monoculture too. Nothing to laugh at!


this wouldn't happen if those programmers did their job.

you can't just "hide" the code by using ioncube and hope that no one will find the exploit.

if code can be encrypted, it can be decrypted. if you don't believe go visit decry.pt.


----------



## zero

"Why would your customers sue you? Your SLA should not promise anything more than the SLA CVPS has for you."

data, money, time you choose


----------



## netnub

ChicagoVPS, treating you as a number, not a name:

6298,

This is a further status update to the recent security breach that ChicagoVPS has experienced. We have successfully restored some nodes, and the vast majority of our VPS customers are online, however we have a small percentage of nodes which still need to be worked on. Some of the nodes we are working on had data loss that we cannot restore. These nodes are LA18, ATL1, ATL4, ATL5. I you on are on one of these nodes you can safely start to rebuild, or open a ticket asking for this months refund.
    
On a positive note, it seems like SolusVM has released a new update in light of the recent incidents ( http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/ ). However, at this time we do not feel comfortable enabling SolusVM access at this minute as we were a victim of their security vulnerabilities two times in the past 7 months. We are evaluating other alternative panels, but at the moment our priority is taking care of our customers and getting the impacted nodes back in working order.
    
Please understand that we have all hands on deck working tirelessly to restore service connectivity for those impacted. Therefore, our ticket response times are affected to allow us to effectively work without anything slowing us down. We apologize for the delayed ticket response times but we assure you we are making progress and working hard to get everything back to normal.

Our goal is to have everything 100% restored tomorrow. Those affected by this incident will recieve compensation.
    
Regards,

ChicagoVPS Team


----------



## Flapadar

zero said:


> My Customers threat with lawsuit.
> 
> ChicagoVPS cant answer the tickets.
> 
> When system up and running ?
> 
> I need net time for system up and running!


Easy solution to that. Set their services to be cancelled at the end of the billing period, tell them "We won't respond any more. Please have your lawyer contact us / our lawyer by snail mail"

And then problem solved.


----------



## johnnyd95

Me and Curtis G are releasing 0day vun for hostbill and whmcs in 2 days on Friday at noon 12pm est. opcorn:

And yes, we can hack engwish


----------



## Reece-DM

johnnyd95 said:


> Me and Curtis G are releasing 0day vun for hostbill and whmcs in 2 days on Friday at noon 12pm est. opcorn:
> 
> 
> And yes, we can hack engwish


Of course you are!


Next you'll be releasing remote root access to cpanel I hope?


Oh hold up that won't happen either your just some kid trying to get publicity in an idiotic delusional way.


----------



## johnnyd95

Reece said:


> Of course you are!
> 
> 
> Next you'll be releasing remote root access to cpanel I hope?
> 
> 
> Oh hold up that won't happen either your just some kid trying to get publicity in an idiotic delusional way.



Releasing remote root access to cPanel, not a bad idea, I'll have to suggest that to Curtis G. Thanks for the idea opcorn:


----------



## epaslv

zero said:


> yes my fault i miss my backups but, i pay for money for service and stability is this cpvs problem not mine. But problem or hack or whatever happed. I 'm wait statement or any respose from cpvs u understand me.


My friend, You take a big risk by using a LEB for mission critical service.

 

1) You have to spend more money and get backups in place with another provider.

2) Then then spend more money again, and backup the backups to a different provider in another country.


----------



## epaslv

zero said:


> I have some backups not at all.
> 
> I'm still waiting but CVPS not respond any ticket or not make statement for us "customers"
> 
> Whats happen right now? I'm in darkness.


This can happen as they are probably in "all-hands-on-deck" mode, trying to recover from a disaster.

I have worked with many companies who are ITIL certified. In the ITIL world they place more importance on "Incident Management" than trying to restore the fault. This is because notifying and updating your customers of the incident is of more value than restoring the fault.

I am not saying it is not important. It just that while you are dealing with a "disaster" you have to take care of the business side of things.


----------



## rbreding

Have read enough of this....have to comment now.

I have a VPS with them and it wasn't affected, passwords changed, moved on.  This VPS is a backup and a test area.  BUT I still have my own backups.

When I take on a new client the first thing that is done is new backups are made a a daily backup system is made.  If someone pays you to "service" them don't you value your time in trying to recover ?  All of my clients have AT LEAST 2 local and 1 remote backup.  If they aren't willing to let me make sure they are protected then I will not take them on as a client.  But the flipside of that is then I am responsible to make sure it is done and tested periodically.

Shame on people for not making sure they have their own backups.


----------



## chronos511

It totally blows my mind the number of people screaming because they didn't have their own back up. All I lost on my VPS was my backup MX config, a ZNC install I had just done and an OTRS install I was just playing around with. I admit I didn't have a backup but ya know what? Had say, the OTRS been a bloody live help desk I would have had a back up. You *never* rely on someone else to do your backups for you even if they say they will. I hadn't done one because quite frankly I don't care one way or the other if I lost it all.

With that said, I'd love to know when I can get back in. I miss my ZNC. ;-)


----------



## upsetcvps

cvps says it makes backups after the last incident.  Turns out they have no clue.  Some nodes don't have backups at all and they're not even sure how old the last backups were for the nodes that do have backups.

If this was the first time this happened to you, cvps, ok maybe I cut you some slack.  But it's not.  And you should know better.  You should learn from your mistakes in the past.  You should know to inform your customers in a timely manner and keep them up to date.  But you don't.  Sure, I didn't expect much from you.  But having used more expensive providers like Linode, I didn't think the premium paid for the name was worth what they provide (and it's not).  But cvps, you managed to surprise me even with my low expectations of you.  You could have handled this worse, but not by much.


----------



## zero

in the *31 hr* waiting ...


----------



## kauffjd3

Still waiting as well.  How can I figure out which node I am on?


----------



## helobye

kauffjd3 said:


> Still waiting as well.  How can I figure out which node I am on?


Very interested to know this too.


----------



## texteditor

I don't give a shit if you are paying $3/mo or $300/mo on a service,

*DON'T RELY ON YOUR PROVIDER FOR BACKUPS, YOU IDIOTS*


----------



## CVPS_Chris

upsetcvps said:


> cvps says it makes backups after the last incident. Turns out they have no clue. Some nodes don't have backups at all and they're not even sure how old the last backups were for the nodes that do have backups. If this was the first time this happened to you, cvps, ok maybe I cut you some slack. But it's not. And you should know better. You should learn from your mistakes in the past. You should know to inform your customers in a timely manner and keep them up to date. But you don't. Sure, I didn't expect much from you. But having used more expensive providers like Linode, I didn't think the premium paid for the name was worth what they provide (and it's not). But cvps, you managed to surprise me even with my low expectations of you. You could have handled this worse, but not by much.


Why are you saying false information? We have backups and know how old they are. As for this happening twice, its because no one listened to me when I said it was a Solus issue the first time. If I was listened to, maybe this would have been found months ago and it would have saved myself, Ramnode, and the other provider from what we are going through.

What do you expect in a 24 hour period? With all the problems we are dealing with its more important to get everyone back online that to write a response every hour saying "We are still working on it". Do you think I am sitting around eating a sandwich laughing at all of this? The answer is no, and this is a very serious matter.

I know what I say will not change your mind, but at least get your facts straight so you dont scare people that do not know better and will listen to you.


----------



## Otakumatic

Let CVPS do their work. God, some people are so impatient....

Also, the supposed "hackers", shut the fuck up.

/mytwocents


----------



## texteditor

CVPS_Chris said:


> As for this happening twice, its because no one listened to me when I said it was a Solus issue the first time. If I was listened to, maybe this would have been found months ago and it would have saved myself, Ramnode, and the other provider from what we are going through.


It's almost as if no one takes you seriously. Wonder why that is


----------



## zero

Anybody know what happend realy on cvps ?

or what now in there ?


----------



## Chankster

You couldn't be more wrong.  In a crisis situation it is still important to keep your customers informed. 



CVPS_Chris said:


> What do you expect in a 24 hour period? With all the problems we are dealing with its more important to get everyone back online that to write a response every hour saying "We are still working on it".


----------



## WelltodoInformalCattle

When customers log into their servers after this episode:


----------



## drvelocity

This is just an epic fail.  Obviously this company had no realistic backup/failsafe system in place for this kind of event despite already having this happen once before.



> After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.
> 
> Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.
> 
> We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.
> 
> Regards
> 
> The ChicagoVPS Team


----------



## zero

@CVPS_Chris Excuse me but you must periodicly give information to customers. I'm your customer but I'm darkness in now. 

Which vps 's live or dead 

or what about the time frame I dont know.

Please give more information about this problem .... Please .....


----------



## Nth

Just a heads up for everyone waiting to get their VPS restored you have to open a ticket to have them to do it. I do see the logic in it as people with backups could restore thier servers faster than cvps can. Personally I don't feel like reuploading 20gigs with my slow upload speed. Even after this as long as they do manage to get my vps back up in a few days (I hope shorter time) I'd still consider it worth the 30 bucks I paid.


----------



## CVPS_Chris

@Zero, the recent email explains it all and should have all the fresh installs up within the next few hours. If you want us to try and restore from our backups, you need to open a ticket.



drvelocity said:


> Obviously this company had no realistic backup/failsafe system in place for this kind of event despite already having this happen once before.


You could not be more wrong, we have backups for all nodes, we just simply cannot make them load any faster then they are, that is why I have decided to change the plan.


----------



## Tux

Amitz said:


> I still do not understand why people who have "important" data on a VPS do not keep own backups. Really. I even have backups of the most unimportant data. If one of my VPS provides goes down, it will take me a max. of 5 hours to be fully operational at another place. And I am just a "hobbyist"... Shame on all "professionals" for not having backups.


I totally agree. When one of my RamNode VPSes got wiped, I thankfully had a copy of the Minecraft world it hosted on another machine.


----------



## zero

@CVPS_Chris I need only data I dont care new vps or something. Which locations affected data loss and Whats the damage on there ?


----------



## orizzle

My machine is finally back up. Looks like the backup they restored was from about a week ago, though. Luckily I made my own backups. Switching to another VPS provider ASAP!


----------



## CVPS_Chris

orizzle said:


> restored was from about a week ago


It clearly states weekly backups.....


----------



## zero

@CVPS_Chris please check your private messages.


----------



## xvtv

Is buf-vps19 going to be up soon?


----------



## zulualpha

CVPS_Chris said:


> It clearly states weekly backups.....


So all the Buffalo nodes that aren't already up should be up in the next hour or so? 

Will new root passwords be emailed out to everyone? 

If customers choose to open a ticket and get their VPS restored from your backup individually, how much longer would that end up taking?


----------



## drmike

> This can happen as they are probably in "all-hands-on-deck" mode, trying to recover from a disaster.


At last check there aren't many hands on board at ChicagoVPS.  The staff is laughably non existent for their user base:

select username from administrators;

+----------+

| username |

+----------+

| vpsadmin | 

| layotte  |

| fabocj40 |

| tleonard |

| adamng   |

| matthew 

 

6 accounts with some padding in there (i.e. fake accounts and a CC backdoor most likely).  layotte, tleonard and adamng are the three admins with tech know how.

 

Three admins to deal with how many customers?

 


select count(clientid) from clients;

+-----------------+

| count(clientid) |

+-----------------+

|            8025 |

+-----------------+

1 row in set (0.00 sec)

 


 

The total number of virtual servers active as of the hack?

 


select count(DISTINCT(vserverid)) from vservers where disabled = '0';;

+----------------------------+

| count(DISTINCT(vserverid)) |

+----------------------------+

|                       9357 |

+----------------------------+

1 row in set (0.07 sec)

 


Too many people put their "eggs" in an already "full fool" basket at ChicagoVPS.  You folks were _buffalooed_ by those crazy low giveaway prices.


----------



## JDiggity

buffalooed said:


> At last check there aren't many hands on board at ChicagoVPS.  The staff is laughably non existent for their user base:
> 
> select username from administrators;
> 
> +----------+
> 
> | username |
> 
> +----------+
> 
> | vpsadmin |
> 
> | layotte  |
> 
> | fabocj40 |
> 
> | tleonard |
> 
> | adamng   |
> 
> | matthew
> 
> 
> 
> 6 accounts with some padding in there (i.e. fake accounts and a CC backdoor most likely).  layotte, tleonard and adamng are the three admins with tech know how.
> 
> 
> 
> Three admins to deal with how many customers?
> 
> 
> 
> 
> select count(clientid) from clients;
> 
> +-----------------+
> 
> | count(clientid) |
> 
> +-----------------+
> 
> |            8025 |
> 
> +-----------------+
> 
> 1 row in set (0.00 sec)
> 
> 
> 
> 
> 
> 
> The total number of virtual servers active as of the hack?
> 
> 
> 
> 
> select count(DISTINCT(vserverid)) from vservers where disabled = '0';;
> 
> +----------------------------+
> 
> | count(DISTINCT(vserverid)) |
> 
> +----------------------------+
> 
> |                       9357 |
> 
> +----------------------------+
> 
> 1 row in set (0.07 sec)
> 
> 
> 
> 
> Too many people put their "eggs" in an already "full fool" basket at ChicagoVPS.  You folks were _buffalooed_ by those crazy low giveaway prices.


That is only 58 people per server if you figure 160 servers.  At least with Adam / Kevin has brought that down from the 100 per server under Jerimiah.


----------



## drmike

24khost said:


> That is only 58 people per server if you figure 160 servers.  At least with Adam / Kevin has brought that down from the 100 per server under Jerimiah.


 

Well, it isn't that simple.  Fabozzi lied about the number of nodes then around the last hack and has lied prior to this hack.

select nodeid from nodes;

= a list of all nodes

High nodeid = 151.

But there aren't 151 nodes 

select count(nodeid) from nodes;

+---------------+

| count(nodeid) |

+---------------+

|           109 |

+---------------+

1 row in set (0.00 sec)

 

 

Of those 109, 5 of them aren't VPS to customer nodes if I remember correctly (perhaps the backup servers?).  Comes down to 104 node servers.


----------



## Aldryic C'boas

> That is only 58 people per server if you figure 160 servers.


160 servers? Hardly.

Total number of nodes



Code:


SELECT COUNT(nodeid) FROM nodes;

Total number of VMs per Node, plus how much "Guaranteed" RAM (in GB) is sold per node


Code:


SELECT nodes.nodeid, nodes.name, COUNT(vservers.nodeid), SUM(vservers.ram)/1073741824 FROM nodes, vservers WHERE nodes.nodeid = vservers.nodeid GROUP BY nodes.nodeid;


----------



## drmike

Who has/can find the list of nodes that were lost/deleted/hosed in this attack?

ATL-VPS1 was one of them.

That server had 317GB of RAM sold on it and 161 virtual servers running on it.


----------



## mnsalem

buffalooed said:


> Who has/can find the list of nodes that were lost/deleted/hosed in this attack?
> 
> ATL-VPS1 was one of them.
> 
> That server had 317GB of RAM sold on it and 161 virtual servers running on it.


Well, im on BUF-VPS19 and its down, so im guessing compromised too


----------



## MartinD

C'mon guys - can we keep this thread on topic?

I know many people think he's an ass but i don't think it's right to be publishing information like that. If people want it I'm sure they can find the DB elsewhere for themselves


----------



## mnsalem

MartinD said:


> C'mon guys - can we keep this thread on topic?
> 
> I know many people think he's an ass but i don't think it's right to be publishing information like that. If people want it I'm sure they can find the DB elsewhere for themselves


Just goin' with the flow 


But really, i see they are working really hard and continuously on it! 


the number of nodes that is up now is much better than what it looked like this morning! According to Pingdom that is.


----------



## leeboof

mnsalem said:


> Just goin' with the flow
> 
> 
> But really, i see they are working really hard and continuously on it!
> 
> 
> the number of nodes that is up now is much better than what it looked like this morning! According to Pingdom that is.


Seriously... I think the exact same amount of servers are down. Didn't they say they weren't restoring unless requested afterwards? What could be taking so long.


----------



## drmike

This is my last tidbit of info so folks understand the scope of the attack from a total victim/client perspective and why restores (even with enough man power) could take eons:


These are the CVPS nodes that reported high downtime in Pingdom and assumed to be nodes where major problems and data loss might have occurred:

NodeName = VPSes on Node  Total RAM Sold
atl-vps1 =  161 VPSes      317GB RAM
atl-vps4 =  122 VPSes      250GB RAM
atl-vps5 = 92 VPSes         197.75GB RAM
buf-vps17 =  100 VPSes   199.375GB RAM
buf-vps19 = 117 VPSes    216.5GB RAM
chi-vps10 = 23 VPSes      18.5GB RAM
chi-vps11 = 31 VPSes      48.875GB RAM
chi-vps12 = 29 VPSes      52GB RAM
chi-vps13 = 11 VPSes      6.75GB RAM
chi-vps14 = 30 VPSes      57.5GB RAM
chi-vps16 = 32 VPSes      37.25GB RAM
chi-vps17 = 71 VPSes      64.49GB RAM
chi-vps18 = 17 VPSes      17.25GB RAM
chi-vps24 = 11 VPSes      9.75GB RAM

= 847 VPSes impacted
 

[SIZE=11pt]LA18[/SIZE]

 is another node where someone confirmed data loss (it isn't in Pingdom monitoring)

la-vps18 = 62  VPSes      92.625GB of RAM

=  909 VPSes impacted


----------



## mnsalem

leeboof said:


> Seriously... I think the exact same amount of servers are down. Didn't they say they weren't restoring unless requested afterwards? What could be taking so long.


Well, in the morning, 3 Atlanta nodes were down .. CHIVPS12 was down .. CHIVPS25 was also down .. besides the 3 BUF nodes still down ... so that is clearly NOT nothing!


----------



## Lanarchy

The email states we will get fresh VPS, but how am I supposed to access or image these with no control panel?

I don't mind waiting, but how can we get fresh VPS with no control panel?


----------



## upsetcvps

CVPS_Chris, on 19 Jun 2013 - 12:38 PM, said:

> Why are you saying false information? We have backups and know how old they are. 

You yourself have stated explicitly that several nodes do not have backups.  At first, you stated it was just some nodes in ATL.  But later you stated there was also an LA node with data loss, which I presume means it did not have backups.  You've also stated that the backups are "at most a week old".  This suggests you do not know the age of the backups though maybe you were just being vague and do know.

 

I am not "saying false information."  If you need me to quote you verbatim on anything, just let me know and I'll dig through your posts here, your posts at LET, and your e-mails.

 

>  As for this happening twice, its because no one listened to me when I said it was a Solus issue the first time. If I was listened to, maybe this would have been found months ago and it would have saved myself, Ramnode, and the other provider from what we are going through.

 

That's your problem.  I deal with you, not Solus.  It's your decision to use their product the way you are using it.  The fact that you were sure Solus had significant issues and didn't take steps to either replace it or put safety measures in is even worse.  At the very least you could have had a sane disaster recovery plan.

 

> What do you expect in a 24 hour period? With all the problems we are dealing with its more important to get everyone back online that to write a response every hour saying "We are still working on it". Do you think I am sitting around eating a sandwich laughing at all of this? The answer is no, and this is a very serious matter.

 

I expect you to immediately notify your customers as soon as you detect an intrusion.  You can be vague at this point.  Once things settle down, you understand exactly what happened and have a plan in place to fix it, you should relay exact details and time-frames to your customers.  You did neither. Some of your customers still have no idea when they will be back up.

 

> I know what I say will not change your mind, but at least get your facts straight so you dont scare people that do not know better and will listen to you.

 

As I said, my facts are straight.  You are the one contradicting yourself.

 

Finally, the grammar in your latest e-mail is atrocious.


----------



## Chankster

CHI59 is definitely still impacted.


----------



## redjersey

buffalooed said:


> This is my last tidbit of info so folks understand the scope of the attack from a total victim/client perspective and why restores (even with enough man power) could take eons:
> 
> These are the CVPS nodes that reported high downtime in Pingdom and assumed to be nodes where major problems and data loss might have occurred:
> 
> 
> NodeName = VPSes on Node  Total RAM Sold
> 
> 
> atl-vps1 =  161 VPSes      317GB RAM
> 
> 
> atl-vps4 =  122 VPSes      250GB RAM
> 
> 
> atl-vps5 = 92 VPSes         197.75GB RAM
> 
> 
> buf-vps17 =  100 VPSes   199.375GB RAM
> 
> 
> buf-vps19 = 117 VPSes    216.5GB RAM
> 
> 
> chi-vps10 = 23 VPSes      18.5GB RAM
> 
> 
> chi-vps11 = 31 VPSes      48.875GB RAM
> 
> 
> chi-vps12 = 29 VPSes      52GB RAM
> 
> 
> chi-vps13 = 11 VPSes      6.75GB RAM
> 
> 
> chi-vps14 = 30 VPSes      57.5GB RAM
> 
> 
> chi-vps16 = 32 VPSes      37.25GB RAM
> 
> 
> chi-vps17 = 71 VPSes      64.49GB RAM
> 
> 
> chi-vps18 = 17 VPSes      17.25GB RAM
> 
> 
> chi-vps24 = 11 VPSes      9.75GB RAM
> 
> 
> = 847 VPSes impacted
> 
> [SIZE=11pt]LA18[/SIZE]
> 
> is another node where someone confirmed data loss (it isn't in Pingdom monitoring)
> 
> 
> la-vps18 = 62  VPSes      92.625GB of RAM
> 
> 
> =  909 VPSes impacted


this makes sense. They sell quite a lot of 2gb vps for $30 to $40year. So buf-vps19 = 117 VPSes 216.5GB RAM = 117 x 30 = $3510 / 12 = $292.5 per month


----------



## nunim

So... this just tells us exactly what we knew already, CVPS nodes are massively oversold and likely are using SSD's as ram.


----------



## imperio

It seems like there is a disclosure campaign ongoing for Colocrossing/Chicagovps.


----------



## Lanarchy

If you're surprised by the fact that VPS are oversold, you're delusional. This almost seems like a hate train. Yes, I get it, yes I understand where everyone's coming from, but this just seems like an anti-CVPS circlejerk at this point.\

I still like knowing the info, like how many containers are on each node, but some of the posts in here are just not necessary.


----------



## redjersey

nunim said:


> So... this just tells us exactly what we knew already, CVPS nodes are massively oversold and likely are using SSD's as ram.


what, you don't do math? they are charging a 2gb vps for only $30-40/year. To make profit they have to put 100-120 vps into one node. What do you expect? 15 2gb vps on a 32gb server??


----------



## jfreak53

Moderator, please clean this up again or close it? I would prefer clean, this thread is for the problems happening not a trashing thread, if they want to trash they can open their own thread for that. Thank you.


----------



## drmike

> You yourself have stated explicitly that several nodes do not have backups.  At first, you stated it was just some nodes in ATL.  But later you stated there was also an LA node with data loss, which I presume means it did not have backups


From reading the various public releases and keeping up a bit on CVPS:

1. Atlanta nodes were not being backed up.

2. Backups *seem* to be on a weekly basis.

Good luck to those of you who weren't self-backing up to another VPS elsewhere.   Your data and VPS if not online by now is very likely GONE.

Those of us that lived through the last attack can attest to fact that it took 3+ days before it was clear many VPSes were lost/gone.


----------



## AnthonySmith

zero said:


> yes my fault i miss my backups but, i pay for money for service and stability is this cpvs problem not mine. But problem or hack or whatever happed. I 'm wait statement or any respose from cpvs u understand me.


You don't pay for financially backed SLA enterprise grade hosting, you pay for basic unmanaged VPS hosting who make it clear they are not responsible for your loss of data, again if you are making promises to the degree that your customers can claim financial compensation from you then you should be using a self healing cloud based solution with a financially backed guarantee or have your own insurance in place.


----------



## drmike

AnthonySmith said:


> You don't pay for financially backed SLA enterprise grade hosting, you pay for basic unmanaged VPS hosting


 

So very true.

For the newbies, redundancy is N+1, which means THREE.  That's right, if you are making money/have paying customers you should have a live with live standby VPS running -- either a cluster or something with low enough DNS TTL's to get people semi-gracefully over to the live server(s) with minimal delay.  But even that is only wise where you have the third backup site.  

By backup here, I don't mean dead data on a storage VPS either.  I mean built, configured, debugged, ready to do business servers/VPS/whatever.

Yes, you can achieve N+1 redundancy with low cost servers.   Many folks are doing it and it works very well.


----------



## shovenose

Every single BetterVPS sign up has been people running from ChicagoVPS. I can be proud of that. But for the price, ChicagoVPS provides the expected service. 

One of my ChicagoVPS-hosted nameservers now has a 1 day uptime but it's not erased and WHM is still accessible. I should probably change the password though.


----------



## xvtv

I have (had) one VPS with ChicagoVPS and another free with Host1Free, I'm glad I didn't bother to do backup between them...

They are still down...


----------



## infinityhosting

My vps is still down and I have not had a reply to my ticket. WAA-970698 I opened it 12 hours ago. I just want my vps back up so I can get back in business like everyone else.


----------



## drvelocity

I love how the owner of the company shows zero remorse for how much pain and agony his customers are suffering through and will continue to suffer through.  Not just counting the losses in sales from the downtime and potential client-loss, there will be thousands or tens of thousands of man hours required to rebuild all of these wiped VPS servers.


----------



## upsetcvps

From http://chicagovps.net/about.html

> What do you're current customers think of you?

hahahaha x2

It's like an 8-year-old wrote this page...

By the way Chris, what do *your* lucky customers win because of *your* 99.9% uptime guarantee?


----------



## MannDude

So, was there any official statement yet on what the cause of their hack was?

Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?


----------



## mnsalem

MannDude said:


> So, was there any official statement yet on what the cause of their hack was?
> 
> Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?


God knows.

Now i'm seeing more nodes going offline on Pingdom ... this is not good :/


Its supposed to go the other way around ...........


----------



## upsetcvps

MannDude said:


> So, was there any official statement yet on what the cause of their hack was?
> 
> Was it the original SolusVM exploit that impacted RamNode as well, or was it something else?



Well Chris blamed solusvm (again) a few comments ago but who knows what that means.


----------



## MannDude

mnsalem said:


> God knows.
> 
> Now i'm seeing more nodes going offline on Pingdom ... this is not good :/
> 
> 
> Its supposed to go the other way around ...........


Woah, can anyone who is a customer confirm if this is true or not: https://twitter.com/christruncer/status/347407782057742336



> "So, @*ChicagoVPS* decided that restoring customer VPSs from their backups is “too time consuming” and is instead telling people to start over."


Could be some data on the backup nodes ( I think there was 4 of them ) could have been wiped to?


----------



## upsetcvps

MannDude said:


> Woah, can anyone who is a customer confirm if this is true or not: https://twitter.com/christruncer/status/347407782057742336
> 
> Could be some data on the backup nodes ( I think there was 4 of them ) could have been wiped to?


yeah the tweet is true.  Here's their latest e-mail (it was posted earlier here but I'll repost):



> After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.
> 
> Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.
> 
> We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.
> 
> Regards
> 
> The ChicagoVPS Team


Mind you, there are *still* people without a vps of any kind, so I can't imagine what the fuck they are doing.


----------



## zulualpha

I'm still without my two VPSs in Buffalo. I've got my own backups, and since they're not restoring backups now I don't know what's taking so long to get the nodes up


----------



## Mun

Cameron Munroe (Munroenet)

After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.

Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.

We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.

Regards

The ChicagoVPS Team


----------



## SeriesN

johnnyd95 said:


> Releasing remote root access to cPanel, not a bad idea, I'll have to suggest that to Curtis G. Thanks for the idea opcorn:


Tits or GTFO.


----------



## jacobsta811

Got the same "we don't really have any backups, or least any automated restore". The only one of my 4 nodes back up at all today was atlanta, and that was up, I changed the password, and shut it down because apt-get update failed. Atlanta was back up again, with the *old* password later, so they reimaged it twice (no tickets in at all right now). Shut it down again because apt-get update and apt-get anything was still failing. God only knows what they are doing. Buffalo, Los Angeles, Chicago all still down. All of these are the 2GB/$40/yr plans.

I do note that however or whatever they are "overselling" does *NOT* matter. All that matters is actual performance of my VPS and any issues it has. While they were running performance was satisfactory and I had no issues. Clearly they have giant issues outside of normal operation, but the amount of overselling/overprovisioning does not matter unless it causes an *actual* impact.


----------



## Francisco

jacobsta811 said:


> least any automated restore


Didn't someone post that they had central backup servers in at least 2 of their locations though? I'm assuming solus has some sort of 'mass restore' option from that?

Francisco


----------



## jacobsta811

Clearly they don't, or it isn't setup right, or they have no backups.  Else we'd be restored by now, 36 hours later, wouldn't we ?


----------



## Nick_A

I don't know about central restore, but regular old ftp backups have to be done one by one in SolusVM. This is why it took me 24 hours to do a few hundred VPSs. Only one can be restored per node at a time.


----------



## drmike

Folks unsure of the status of their VPS should submit tickets to CVPS directly.  I know their staff is insanely flooded and probably will have a good wait time for a response.

Folks are chattering a bit on Facebook: https://www.facebook.com/chicagovps/posts/683328475017314

If you lost data, it is your fault.  You are responsible for your own data.  Provider backups aren't sufficient ever and often take eons to have restored.  They are a last resort sort of thing.

If you feel slighted by the uptime guarantee, file with them for a credit.  ChicagoVPS is offering at least a month credit to those impacted.

apt-get problems = more problems.   Post the error and we'll help you debug.

Overselling = extreme.  Problem with the overselling more than anything is overselling relative to staff resources.  3 admins doing 8 hour shifts each means 7 day work weeks and folks always on call.   A company with 8k-9k customers should be better staffed - at minimum 5-7 technical administrators.   When you buy a VPS that costs more than $7 for 2GB, having redundancy, security and staff should be places that cause the bottom line to increase.  Think about that when shopping for your new provider and quit being a cheapskate solely.

Finally, it doesn't seem this hack will be disclosed or determined what occurred.  The one in November went the same way.   Finger pointing at Solus then and this time.   Since the SolusLabs folks failed to do anything in public about the accusations in November/February, we can only assume Solus had bugs and themselves couldn't figure out the vulnerability.   

ChicagoVPS didn't hack their system and destroy data.  A criminal did that.  CVPS is a victim as much as the customers who have lost their data and services.


----------



## jacobsta811

apt-get was throwing errors that it couldn't resolve the hostnames of the mirrors. Could be a DNS issue with whatever DNS they have setup by default, or it could be an issue with the setup in the restored slice. Normally it works out of the box in a reimage. I could have spent time trying to solve it, but it was an indication to me that it isn't ready for me to screw with, so I just shut it down (and obviously I can't restart it without solusVM or opening a ticket).

RE:Overselling. Overselling vs # of support staff is a big problem as we can see now, I agree. But people here were also complaining about overselling on specific nodes, and I disagree about that, for any provider - as long as the performance remains adequate for what I need 100% of the time, it isn't and shouldn't be my or your concern how they have the server provisioned/sold.

I am not worried about lost data, as I maybe lost 2 blog posts or something. More time will be lost setting up the boxes again than anything. The bigger issue is having a place to *put* my data. Most of the cheap hosts use SolusVM so I am pretty much out of luck right now. I'm thinking I probably will get a node at South Bend VPS and setup there. I had a full mirrored setup using IP failover, unison, and MySQL replication with geographic separation of the servers in Atlanta and Buffalo, but both were on ChicagoVPS. Clearly an issue, but it seems like I would have needed not only two providers but two different *control panels* to really be safe.

Edit to add: I run my mail server out of Linode, but they are kind of pricy and the disk space is really small on the reasonably priced options for use for my personal websites/blogs/etc that are not really that critical.


----------



## upsetcvps

jacobsta811 said:


> The bigger issue is having a place to *put* my data. Most of the cheap hosts use SolusVM so I am pretty much out of luck right now. I'm thinking I probably will get a node at South Bend VPS and setup there. I had a full mirrored setup using IP failover, unison, and MySQL replication with geographic separation of the servers in Atlanta and Buffalo, but both were on ChicagoVPS. Clearly an issue, but it seems like I would have needed not only two providers but two different *control panels* to really be safe.


my other vps is over at buyvm and it works well enough for me.  It's also cheap but at least they do use stallion instead of solusvm


----------



## drmike

jacobsta811 said:


> but both were on ChicagoVPS


 

There weren't too many hosts smacked by this.  RamNode was the highest profile provider around here and while it took time, I think they rescued everything.

There were something like 1000+ people who had more than one VPS with CVPS.   So, that practice is roughly 10% of their total VPSes deployed.

There are companies who aren't based on SolusVM.  There is an ongoing thread about that on here.   Backupsy comes to mind and BlueVM and BuyVM (I think).

You can cheaply create failover capability with 3 VPSes (different providers) and Rage4 DNS (free depending on your use).  For database security do cron jobbed mysqldumps and scp those off to a remote backup storage like Backupsy.  Ditto for source files, rsync them over to Backupsy.  Synchronizing the database from the failed over to node is another complex story, but minimal impacted users in that scenario.


----------



## zulualpha

CVPS_Chris said:


> @Zero, the recent email explains it all and should have all the fresh installs up within the next few hours. If you want us to try and restore from our backups, you need to open a ticket.


Chris, can we get an update, since 10 hours have passed & it looks like fewer nodes are up now than before. Are you still doing fresh installs?


----------



## MannDude

zulualpha said:


> Chris, can we get an update, since 10 hours have passed & it looks like fewer nodes are up now than before. Are you still doing fresh installs?


Looks like 15 nodes are down right now: http://stats.pingdom.com/jzrszp4wfu79

Though I don't think their Pingdom monitors all of them. The leak revealed they had 109 nodes, which is strange considering the hack back in November revealed about 45-50 nodes (I believe). That was after being in business for many years. So after that hack, their business doubled in less than 9 months? That doesn't make sense. I expect after all this is resolved, that they'll have 250 nodes in 3 months.


----------



## MannDude

This is interesting: https://twitter.com/FrantechCA/status/347533537014075392



> @stormandsong Board arrived this morning. ChicagoVPS got hacked so the DC has been busy dealing with that all. They told us 'a few hours'.


CC must be busy with getting LET working halfway properly, and dealing with CVPS. Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.


----------



## SeriesN

I am surprised no one here used Chris's formula and offered "refugee" coupons. 

Peer respect! This is another thing that sets this forum apart.


----------



## Aldryic C'boas

MannDude said:


> Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.


I'm getting pretty close to the point of becoming vocal about it, aye.


----------



## Mun

I'm sorry but people are just idiots.

Mun


----------



## leeboof

I really hope they update later tonight or early tomorrow morning as to what the deal is. It went from servers should be up in 24 hours to we had an issue restoring so everything will be fresh and running today to no update.

At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.


----------



## MannDude

Has anyone who had a server with dataloss been brought back online from a CVPS restored/maintained backup yet? I'm curious if there are customers who earlier posted issues that are now resolved.


----------



## srichter

leeboof said:


> At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.


The problem being "Well we probably already lost a ton of customers so let's just take a fucking break for a bit."


----------



## drmike

MannDude said:


> Sucks other paying customers of CC is getting delayed because the datacenter staff is more focused on helping their friend out.


 

Here, I'll say it, hasn't BuyVM had parts on site at Colocrossing since before end of the business day?  By my clock, 8 hours or so right?


----------



## drmike

*Remote hands:*

Remote *Hands* (Everything not covered above) $125/Hr

$125/hr    x   24 hours = $3000 per 24 hours 

$3000 per 24 hours    x   number of admins allocated = $6k for 2  $9k for 3....

Minus any discount 

Average income per VPS can't be very high based on pricing.  Probably $3.50 per VPS.

$3.50 x 9000 VPSes = $31,500 a month.

3-4 days of round clock outsourced to CC hands = negative income month.


----------



## Francisco

buffalooed said:


> Here, I'll say it, hasn't BuyVM had parts on site at Colocrossing since before end of the business day?  By my clock, 8 hours or so right?


Motherboards isn't something we kept spare in NY since we have so few KVM nodes there.

We missed the "nest day window" on amazon by a few hours so we lost a whole day.

Francisco


----------



## XFS_Duke

leeboof said:


> I really hope they update later tonight or early tomorrow morning as to what the deal is. It went from servers should be up in 24 hours to we had an issue restoring so everything will be fresh and running today to no update.
> 
> At least tell us what the problem is and give us a realistic expectation of downtime. There is no way by now they don't know what exactly the problems are to share with us.


They are restoring accounts now. They are setting up new VPS accounts for most people. I know this for a fact, so just take a chill pill. There are a lot of servers that they weren't able to just "restore". If you want yours restored, submit a ticket and they'll get to it. If you want your VPS back up and you have backups yourself, then maybe we can work something out. For now, they're running through each node recreating accounts. Just give them time...


----------



## drmike

Sad that folks not in the know and just customers feel ignored, abandoned, etc.







Cheysser Estrella Valdez Update please

3 hours ago
















 



Drew Read Update please...!

3 hours ago














 



Joel DeVenney This is CRAZY!!! Two days and counting, going on 3 now and their last email they promised to have everything back online today!!!

3 hours ago















 



Christopher Breen This is ridiculous. No response to tickets, no restoration of any kind to remaining hosts... Seriously amateur hour...

about an hour ago via mobile


----------



## XFS_Duke

buffalooed said:


> Sad that folks not in the know and just customers feel ignored, abandoned, etc.
> 
> 
> 
> 
> 
> 
> 
> Cheysser Estrella Valdez Update please
> 
> 3 hours ago
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Drew Read Update please...!
> 
> 3 hours ago
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Joel DeVenney This is CRAZY!!! Two days and counting, going on 3 now and their last email they promised to have everything back online today!!!
> 
> 3 hours ago
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Christopher Breen This is ridiculous. No response to tickets, no restoration of any kind to remaining hosts... Seriously amateur hour...
> 
> about an hour ago via mobile


They should be getting emails... Unless their email is being hosted on the VPS... Now, if I were Chris I'd post this on their Facebook as well, if it wasn't already, didn't check...

EDIT: Doesn't look like they did, I posted, maybe someone will read it.


----------



## srichter

> I just want to start off by saying thank you to everybody and their patience. I know this situation has been very frustrating and long, and I can assure you we are doing our best. We are still restoring VPS' via support ticket request. While doing this, we have noticed that a small percentage of the backups were corrupted after being restored. Those of you that are wondering why it’s taking so long for us to get to your ticket, I wanted to clear that up. There are a lot of customers impacted and this is a very timely process and the ChicagoVPS team is going through them very carefully.
> 
> 
> I want to state again that there will be compensation for all clients affected by the compromise. We would like to thank you again for your patience, and we are getting closer to getting everything back in working order.
> 
> 
> Thank you,
> 
> The ChicagoVPS Team


----------



## upsetcvps

ramnode, out of curiousity, how many vps went down for you and how long did it take you to get them back up (also if you don't mind saying, how many people worked on it and what sort of recovery procedure did you follow)?  I'd like to compare with cvps and maybe they can even get some tips from you.


----------



## infinityhosting

infinityhosting said:


> My vps is still down and I have not had a reply to my ticket. WAA-970698 I opened it 12 hours ago. I just want my vps back up so I can get back in business like everyone else.


NOw 24 hours and no response. I had to get a temporary hosting solution through another vendor. I need my vps up.


----------



## kauffjd3

Still down and no responses to my tickets. 

I started rebuilding on another host.  I completely understand cvps is a victim, but they are the company that are providing a service which failed. 

My friends just came up with all data.  I guess there is still hope. I don't know at this point.


----------



## jacobsta811

MannDude said:


> .... So after that hack, their business doubled in less than 9 months? That doesn't make sense..


They dropped their prices significantly - did they ever sell 1yr/2GB RAM/50-100GB disk for $40 before ? They had a *hugely* popular thread on slickdeals.net which is an enormous board and where I found them - although I knew about LowEndBox and VPSBoard before that.


----------



## upsetcvps

cvps is using magnets to re-image my vps by hand; it's the only explanation I can think of


----------



## leeboof

upsetcvps said:


> cvps is using magnets to re-image my vps by hand; it's the only explanation I can think of


Either that or the "small percentage" of corrupted (or non-existent) backups is really all the servers still not online.


----------



## Otakumatic

I know how to respond to CVPS using magnets:






YES, IT'S A MEME, GOOGLE IT.


----------



## jer

Signed up to post, wanted to note for others:

The CVPS Email makes it sound like if you don't want a restore, don't open a ticket, and you'll get a fresh image (how they'd know what image I don't know).

That option sounds good as I keep backups of backups, and use my own script to quickly customize the server on reimage, so I've been waiting.

I've been seeing chi-vps10 as up in Pingdom, but my VPS doesn't ping on either IP, and there's no route to host for SSH.

I reluctantly opened at ticket, as I know they're busy.. asking for a reimage, a random strong pass, and specifying the OS.

So for others info - just because a node appears up in Pingdom doesn't mean your VPS is there.


----------



## Lanarchy

My nodes are all up except for ATL which are both up, but inaccessible.


----------



## XFS_Duke

wow, it really amazes me how many of you cannot read...

Their SolusVM master is back up, is has all of your information in it. Such as IP addresses, OS you had installed and so forth.

What they are doing is manually recreating each VPS in SolusVM, and what I mean by that is going into each accoung in SolusVM and choosing the correct IP for your account, choosing the correct OS based on the information they have and then clicking REINSTALL. This is going to get you back online. They won't email each and every single customer right when they do it for them, as it'll make the process longer. They are sending out email updates. They are not responding to support tickets right now, I think they said that in an email already, unless it is an emergency.

I think some of you are here just to stir up more BS than actually being upset about this.

You can also request them to do a restore of your VPS, once everything is all said and done, you SHOULD be back to normal how you were before someone decided to stop spanking their winkie for a little while and cause this whole thing.

Just have patience. It is a lot of nodes to go through and a lot of time recreating each one. If you have a better way, let them know. Maybe you can give them some tips that they don't have, otherwise, the bashing because you can't read really isn't helping the situation.

Just my 2 cents


----------



## leeboof

XFS_Duke said:


> wow, it really amazes me how many of you cannot read...
> 
> Their SolusVM master is back up, is has all of your information in it. Such as IP addresses, OS you had installed and so forth.
> 
> What they are doing is manually recreating each VPS in SolusVM, and what I mean by that is going into each accoung in SolusVM and choosing the correct IP for your account, choosing the correct OS based on the information they have and then clicking REINSTALL. This is going to get you back online. They won't email each and every single customer right when they do it for them, as it'll make the process longer. They are sending out email updates. They are not responding to support tickets right now, I think they said that in an email already, unless it is an emergency.
> 
> I think some of you are here just to stir up more BS than actually being upset about this.
> 
> You can also request them to do a restore of your VPS, once everything is all said and done, you SHOULD be back to normal how you were before someone decided to stop spanking their winkie for a little while and cause this whole thing.
> 
> Just have patience. It is a lot of nodes to go through and a lot of time recreating each one. If you have a better way, let them know. Maybe you can give them some tips that they don't have, otherwise, the bashing because you can't read really isn't helping the situation.
> 
> Just my 2 cents


I think everyone would have more patience if they actually put put updates like this, it's not that hard.

9am: working on restoring node 13

10am: still working on node 13, 50 accounts to go

11am: almost done with node 13, node 54 next

11:30am: node 13 complete, working on node 54..

It's not that hard and would go a long way to making customers happy and seeing progress.


----------



## XFS_Duke

leeboof said:


> I think everyone would have more patience if they actually put put updates like this, it's not that hard.
> 
> 9am: working on restoring node 13
> 
> 10am: still working on node 13, 50 accounts to go
> 
> 11am: almost done with node 13, node 54 next
> 
> 11:30am: node 13 complete, working on node 54..
> 
> It's not that hard and would go a long way to making customers happy and seeing progress.


I see your point... But they're doing their best to limit the time it takes to get this done. Just have patience, it'll get worked out.


----------



## upsetcvps

XFS_Duke said:


> What they are doing is manually recreating each VPS in SolusVM, and what I mean by that is going into each accoung in SolusVM and choosing the correct IP for your account, choosing the correct OS based on the information they have and then clicking REINSTALL. This is going to get you back online. They won't email each and every single customer right when they do it for them, as it'll make the process longer. They are sending out email updates. They are not responding to support tickets right now, I think they said that in an email already, unless it is an emergency.


I really hope they are not actually clicking away in a gui vps by vps.  I'm not sure if that's much better than magnets.  I should start my own hosting business.  I can call it "notcvps" or "oppositeofcvps" maybe.


----------



## jfreak53

Though I agree that you can only do what you can do when you can do it and no amount of complaining is going to change that, it would take Chris 2 seconds to post to twitter every hour an update of some sort. This would keep customers very very happy in the long run, damage control is one of the first thing the military teaches to new Officer recruits for a reason


----------



## drmike

jacobsta811 said:


> They dropped their prices significantly - did they ever sell 1yr/2GB RAM/50-100GB disk for $40 before ?


 

No doubt, CVPS has been dropping their prices like crazy since the tail end of 2012.

November 25, 2012, they first issued a limited offer:

*Enterprise $30/year*

2048MB Dedicated Ram
50GB Diskspace
2TB Bandwidth

$2.50 a month on that.

Pricing had been pretty much $7-7.95 a month for said plan in 2012.

$7 a month in 2011, got you their 1GB RAM plan.


----------



## jer

I think most of us do have patience - and understanding. Most of us are probably in the industry in one way or another, and aware of the time it would take.
 



'XFS_Duke said:


> wow, it really amazes me how many of you cannot read...


I stopped reading after that. I did skim though...

You mentioned something about how they know the OS etc to reimage with. I'm sure they do, but it didn't say that anywhere in the email I got. So your effort to illistrate how we don't read is without effect. Did you read the notice?
 



'XFS_Duke said:


> wow, it really amazes me how many of you cannot read... ...bashing because you can't read... ...I think some of you are here just to stir up more BS...


Who stirs what? You said we don't read. [ ] ...Just thought I'd point out your bashing irony.

I admit I intintionally didn't thoughly read your post, it honestly sounded like it was going to be a rant. I did thoughly read the CVPS notifications however.

I don't think my previous post had any BS, so I'm assuming your talking about 'talking about it', or you're talking someone else.

If you were talking about my previous post - where do you suggest we go to talk about the issue? ...If not a public Forum, with a thread about the company having to do with the issue? Just want to make sure I signed up in the right place.

I don't expect updates on the hour, I'll check myself. I am patient, at the 3 day Outage level.

I do expect what I paid for, have understanding for issues, and haven't bashed anyone. I've been with CVPS for over 2 years I think, through their ups and downs, and have stayed. I'm one of those "small business supporter" guys. I don't think that's BS, "provider".

Don't bother responding with answers, the questions were more to help you see your irony. Your 2 cents has no actual worth to me.


----------



## XFS_Duke

jer said:


> I think most of us do have patience - and understanding. Most of us are probably in the industry in one way or another, and aware of the time it would take.
> 
> I stopped reading after that. I did skim though...
> 
> 
> You mentioned something about how they know the OS etc to reimage with. I'm sure they do, but it didn't say that anywhere in the email I got. So your effort to illistrate how we don't read is without effect. Did you read the notice?
> 
> Who stirs what? You said we don't read. [ ] ...Just thought I'd point out your bashing irony.
> 
> 
> I admit I intintionally didn't thoughly read your post, it honestly sounded like it was going to be a rant. I did thoughly read the CVPS notifications however.
> 
> I don't think my previous post had any BS, so I'm assuming your talking about 'talking about it', or you're talking someone else.
> 
> 
> If you were talking about my previous post - where do you suggest we go to talk about the issue? ...If not a public Forum, with a thread about the company having to do with the issue? Just want to make sure I signed up in the right place.
> 
> I don't expect updates on the hour, I'll check myself. I am patient, at the 3 day Outage level.
> 
> I do expect what I paid for, have understanding for issues, and haven't bashed anyone. I've been with CVPS for over 2 years I think, through their ups and downs, and have stayed. I'm one of those "small business supporter" guys. I don't think that's BS, "provider".
> 
> Don't bother responding with answers, the questions were more to help you see your irony. Your 2 cents has no actual worth to me.


Actually, I know what they were doing. I watched it on Team Viewer. My business partner Brian was helping them out yesterday. So, that is how I know. I was supposed to help them out, but I didn't get home in time. So, thank you mr. 2 posts. There was no irony in my questions as I know what they're doing. So, my post was to alert people that they are overracting. Simple as that. Just give it time.

If it's simple and you can do better, by all means, buy a domain, a server, build a design and setup everything as well as put in the money to get accounts that you need in order to run your business and then do it yourself.


----------



## upsetcvps

XFS_Duke said:


> If it's simple and you can do better, by all means, buy a domain, a server, build a design and setup everything as well as put in the money to get accounts that you need in order to run your business and then do it yourself.


How much of an initial financial investment would it take?  Since I am mildly serious.


----------



## XFS_Duke

upsetcvps said:


> How much of an initial financial investment would it take?  Since I am mildly serious.


That would all depend on how you want to look, what options you want to provide for your customers and so forth. Probably around $1000 to start. That gets you all the "options" you need to start. But like I said, it all depends on how you want to look to your customers. If you buy your own servers instead of leasing them, then you'll look at atleast 4x that number.

But how about we don't hijack this thread anymore. If you have questions for me, send me a PM....


----------



## Aldryic C'boas

> I think everyone would have more patience if they actually put put updates like this, it's not that hard.
> 
> 9am: working on restoring node 13
> 10am: still working on node 13, 50 accounts to go
> 11am: almost done with node 13, node 54 next
> 11:30am: node 13 complete, working on node 54..





> it would take Chris 2 seconds to post to twitter every hour an update of some sort


Some of you folks might not appreciate exactly what is going on at the DC right now, and just how hectic things are. When we moved to Vegas, Francisco and I were both up for well over 30 hours straight (with a 10 hour drive): elbow-deep in hardware trying to get things working again, public communications was honestly the absolute last thing on our minds, and to be fair the only reason we remembered to update as often as we did was because we were primarily using IRC to communicate with Anthony, and he'd remind us to update everyone else.

Our issues in Vegas were minor compared to this.. if the SQL count is right, you're talking over 100 nodes apparently trashed. This isn't a case where they're just sitting at their desks, drinking coffee and playing with GUI interfaces. The guys working to resolve this are very likely having to rip nodes apart to test hardware, debug the existing problems, get things settled, all while also trying to juggle 1) The Solus vulnerability, and 2) checking to make sure someone didn't leave a backdoor on a node or something. All of this taking place while under an INSANE amount of stress, which is going to impact concentration and reasoning. If someone is buried in terminal after terminal trying to recover these nodes, they're not going to stop and think "Oh, guess I should go hit up Twitter to remind folks that we're obviously not just leaving things where they fell". In a normal situation, sure, you have a chance to take a breather, remind yourself of other things to do, and so forth. In a hellish situation like this? You're lucky if the poor saps remember to eat, let alone keep track of how much time has passed and so on.

Yes, regular updates are generally the decent thing to do, so that affected clients can keep up with what's going on. But FFS folks, don't presume this is just walk-in-the-park situation where they're just deciding "eh, screw it, no need to update anyone"... at least try to be a bit considerate of what they're going through at the moment.






> damage control is one of the first thing the military teaches to new Officer recruits for a reason


_Offtopic_: They should concentrate more on teaching recruits to show more respect to seasoned NCOs, nevermind how much they paid for the butterbar. I _HATED_ dealing with ROTC-Nazis, especially when they wanted to take out some stupid complex on the sols without us sergeants being able to intervene.


----------



## earl

upsetcvps said:


> How much of an initial financial investment would it take?  Since I am mildly serious.


Anyone with $100 can be a summer host!


----------



## Chankster

Aldryic C said:


> Some of you folks might not appreciate exactly what is going on at the DC right now, and just how hectic things are. When we moved to Vegas, Francisco and I were both up for well over 30 hours straight (with a 10 hour drive): elbow-deep in hardware trying to get things working again, public communications was honestly the absolute last thing on our minds, and to be fair the only reason we remembered to update as often as we did was because we were primarily using IRC to communicate with Anthony, and he'd remind us to update everyone else.
> 
> 
> ...
> 
> 
> Yes, regular updates are generally the decent thing to do, so that affected clients can keep up with what's going on. But FFS folks, don't presume this is just walk-in-the-park situation where they're just deciding "eh, screw it, no need to update anyone"... at least try to be a bit considerate of what they're going through at the moment.


No, regular updates are not "decent" they are required.  It is a standard part of damage control and only amateurs would think differently.  The easiest way to help manage the influx of tickets is to keep your customers informed from the very beginning.  I cannot emphasize this enough: Keeping your customers in the dark will do FAR more damage than the "hack" itself.


----------



## XFS_Duke

earl said:


> Anyone with $100 can be a summer host!


Yep.. But, we don't want summer hosts. Rent a server somewheres and put your data on your own dedicated server.


----------



## earl

XFS_Duke said:


> Yep.. But, we don't want summer hosts.


Yes.. but LEB without summer host is like fries without gravy.. and some of the drama we get it's like ketchup on top! lol


----------



## XFS_Duke

earl said:


> Yes.. but LEB without summer host is like fries without gravy.. and some of the drama we get it's like ketchup on top! lol


Haha yea... Too bad LEB doesn't post offers from only reputable hosts... They pick and choose who they post...


----------



## Aldryic C'boas

Chankster said:


> It is a standard part of damage control and only amateurs would think differently.



Alright, not to be a dick about this, but if you want to nitpick on words, fine... when exactly have you gone through a situation like this?  What relevant experience do you have regarding this level of meltdown?  When's the last majorly stressful disaster you went through where you were able to keep a cool head and not let the overwhelming gravity of the situation seriously skew your mental faculty?

You speak of "amateur", and yet show no evidence that you are anything to the contrary.

No, I'm not defending that there seems to have been little update; if I were a client, I'd be pissed about being in the dark as well - I'm simply providing a little bit of insight as to what a situation like this is like for a provider for those of you that have absolutely no clue, and seem to think it's as simple as "Press button, restore node, run to twitter".  Perhaps instead of just being indignant, you could step back, be honest, and tell yourself "Wow, that's one hell of a situation they got shoved in... I honestly couldn't say if I could do it better since I've never been thrown under the bus like that.  Yeah, it really sucks that there might not be backups or an ETA yet;  but it is what it is, and my yelling won't magically make things go faster".


----------



## jer

Aldryic C said:


> When we moved to Vegas, Francisco and I were both up for well over 30 hours straight (with a 10 hour drive): elbow-deep in hardware


Offtopic: Wanted to add that a company I worked for did the same thing - moved boxes from CA in the back of a truck to their 2nd DC in a different state. There was an issue with adult related sites not having the right licenses for the new state, and the customers didn't have enough notice to correct it. An after issue that made an already hetic issue worse with the 2nd DC unable to handle the new electric load.

On Topic: I understand they're up to their elbows, that's why I haven't bugged 'em. I also don't expect updates each hour if an outage is known. I did expect my vps to be responding after a few hours of the node being up. When it wasn't, I thought I'd sign up and post to let others know their vps may not be up even if the node reports it is.



Aldryic C said:


> show more respect to seasoned NCOs


Offtopic: We were all frineds through the ranks in my shop, even had female flight commanders twice. Most of us lived off base and hung out together. Rotc guys rarely joined, but were always funny to watch.


----------



## drmike

Thanks for some insight @XFS_Duke.  Didn't realize they were splurging for outside help.  Uggh!  That's, umm, well, bad.

The updates are mandatory for customer sake.  Again, there is a reason why a 2GB VPS isn't $2-$7 a month, unless something is wrong (i.e. little to no staff behind the product)

Normal everyday people, the average customer doesn't care why something isn't working.  Customers get mighty miffed when a big storm rips through a town downing utility lines.  A day, maybe two and that's all the tolerance people have.  Think we are at 

4 days now.

To that point:



> Shahyan Aly I wish I could do this in my line of work. Just say "fuck it; i screwed up, and I ain't responding to shit!" Although in retrospect, they never did say they screwed up; CVPS blamed SolusVM. I don't care about SolusVM; I pay you and when stuff like this happens, it's your problem that you're using SolusVM. /endrant


----------



## Aldryic C'boas

jer said:


> I did expect my vps to be responding after a few hours of the node being up.


Yeah, that aspect is a bit worrying.  And honestly, without administrative experience themselves clients don't understand how something like that would happen.

Just to help folks understand (and this is just an example, it may not be what's happening with those nodes at all):  just because an OpenVZ node is online doesn't mean the VPSes on it automatically are.  Think of the VPSes like programs you have installed in Windows - if certain config files are missing, the programs may not run at all.  It's very likely that they got the node online, and there are simply some VPSes who's "config files" were trashed/damaged, and need to be repaired before that VPS can function again.


----------



## Chankster

Aldryic C said:


> Alright, not to be a dick about this, but if you want to nitpick on words, fine... when exactly have you gone through a situation like this?  What relevant experience do you have regarding this level of meltdown?  When's the last majorly stressful disaster you went through where you were able to keep a cool head and not let the overwhelming gravity of the situation seriously skew your mental faculty?
> 
> You speak of "amateur", and yet show no evidence that you are anything to the contrary.
> 
> No, I'm not defending that there seems to have been little update; if I were a client, I'd be pissed about being in the dark as well - I'm simply providing a little bit of insight as to what a situation like this is like for a provider for those of you that have absolutely no clue, and seem to think it's as simple as "Press button, restore node, run to twitter".  Perhaps instead of just being indignant, you could step back, be honest, and tell yourself "Wow, that's one hell of a situation they got shoved in... I honestly couldn't say if I could do it better since I've never been thrown under the bus like that.  Yeah, it really sucks that there might not be backups or an ETA yet;  but it is what it is, and my yelling won't magically make things go faster".


I work in a similar technology service field and I completely understand how difficult it can be to recover from issues.  However, in our business we notify customers as soon as we detect any customer impacting issues.  The more serious the issue, the more frequent the updates (regardless of the amount of information we have uncovered).  Our customers understand that there can be major situations that are started by things outside of our control but it is still no excuse for us to leave them in the dark.


----------



## Tactical

Personally this just they way i feel and it doesnt go for anyone else but dam its only a VPS . Dam ppl we got kids that starve at night. People complaining about something so MINOR! You paid money i understand that but they are doing what they can. They are only HUMAN! If the VPS  was hosting something so big like a business website or it was making your money. Well you should of planned some kind of redundancy! Peace Out cause its summer time summer time!


----------



## Tactical

Chankster said:


> I work in a similar technology service field and I completely understand how difficult it can be to recover from issues.  However, in our business we notify customers as soon as we detect any customer impacting issues.  The more serious the issue, the more frequent the updates (regardless of the amount of information we have uncovered).  Our customers understand that there can be major situations that are started by things outside of our control but it is still no excuse for us to leave them in the dark.


Didn't they update everyone via email and their website. What more you want them to do? Pull a bunny out of their a$$. It is just a bunch of nit picking. So lets enjoy the day and move on!


----------



## kauffjd3

still down.. opcorn:


----------



## Chankster

SgtZinn said:


> Didn't they update everyone via email and their website. What more you want them to do? Pull a bunny out of their a$$. It is just a bunch of nit picking. So lets enjoy the day and move on!


The first email update I received was ~12 hours after my VPS went down.  There has been 1 twitter/facebook update which was ~10 hours after and the client area notice went up around the same time.  There have only been 3 additional email updates in the nearly 60 hours since the beginning.  However, the entire time this has been going on their "Network Status" page reports no incidents.


----------



## drmike

SgtZinn said:


> Peace Out cause its summer time summer time!


 

*Summer... Host.*

Tee hee... Summer time and a host that has been around for years that acts like a summer host 

*Mighty Big Fall*

I remember a thread where CVPS_Chris jumped on @Prometeus and talked all sorts of smackola.  That thread and some other mouthing about being the biggest VPS company in this segment to an equine host.   How the mighty fall and the data shows lies.

*Contingency Planning:*

I wonder how other folks would recover from something like this?  What steps are reputable providers taking to deal with backups on a _regular _basis?   How could you mass migrate backups to new nodes?  Think about it providers.  Get your contingency plans in place and stand out from the pack.

*Social Media and Keeping YOUR Customers*

Love social media and all the power to quickly say something to everyone listening.  Too bad folks in this situation can't find a human, a secretary or the janitor to post to such.  They never seem to have issues pushing the cheap snake oil out via such though  Buy Buy, buy our sh!t!

*Minor Complaints or Minor Threat*

@SgtZinn said this is all minor compared to real world problems and he's right.  

The big butt is many customers lack backups and too many aren't tech-knowledgeable enough to setup their own VPS again from scratch.  People have hours / outsourced / paid others to config and get working.  Many fail to document things adequately to replicate what they had.  These folks are in a world of pain right now.


----------



## jer

Aldryic C said:


> Yeah, that aspect is a bit worrying. And honestly, without administrative experience themselves clients don't understand how something like that would happen.


Yep, that's why I opened the ticket (after several hours of the box reporting up). There may be a problem and they may not be aware if the host is reporting up. I'm also not expecting them to even see the ticket for hours.


----------



## Aldryic C'boas

> Our customers understand that there can be major situations that are started by things outside of our control but it is still no excuse for us to leave them in the dark.


You're 100% correct. If I came across as defending the lack of updates, my apologies for the miscommunication (pretty much anyone that knows Chris and I knows that either one of us defending the other would likely result in hell freezing over ). I was more giving a experienced viewpoint for the clients that don't realize what all a situation like this entails. Aye, for sure there should be more updates (I'm not one of their clients, so I have no clue about the frequency on notifications going on), but maybe now some of them at least know there's more to the situation than them simply being left in the dark for no reason. My posts were definitely more for the clients' sake than anything.


----------



## jer

buffalooed said:


> The big butt is many customers lack backups and too many aren't tech-knowledgeable enough to setup their own VPS again from scratch. People have hours / outsourced / paid others to config and get working. Many fail to document things adequately to replicate what they had. These folks are in a world of pain right now.


I thought I'd read more from those folks in this thread. If they didn't know to have backups, they do now. If their projects were commercial, that's a hard lesson..


----------



## Chankster

Aldryic C said:


> You're 100% correct. If I came across as defending the lack of updates, my apologies for the miscommunication (pretty much anyone that knows Chris and I knows that either one of us defending the other would likely result in hell freezing over ). I was more giving a experienced viewpoint for the clients that don't realize what all a situation like this entails. Aye, for sure there should be more updates (I'm not one of their clients, so I have no clue about the frequency on notifications going on), but maybe now some of them at least know there's more to the situation than them simply being left in the dark for no reason. My posts were definitely more for the clients' sake than anything.


Exactly.  Unless you notify your customers they have no idea what the situation entails and what timeline they should expect.


----------



## upsetcvps

Chankster said:


> The first email update I received was ~12 hours after my VPS went down.  There has been 1 twitter/facebook update which was ~10 hours after and the client area notice went up around the same time.  There have only been 3 additional email updates in the nearly 60 hours since the beginning.  However, the entire time this has been going on their "Network Status" page reports no incidents.


Yes, and Chris was posting here and at LET before even sending out the first e-mail.  He could easily have put a notice on the client page or on twitter.  And these e-mails they are providing are just too vague.  I have no idea if I'll be up soon or only next month.  They don't need to update me every hour, but actually give me some useful information with the updates.  I'm glad you're aware of the issue, that's a good start.  But when am I going to be able to use my vps again?


----------



## infinityhosting

8230


This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.


We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.


Regards


The ChicagoVPS Team


----------



## leeboof

infinityhosting said:


> 8230
> 
> 
> This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.
> 
> 
> We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.
> 
> 
> Regards
> 
> 
> The ChicagoVPS Team


I wonder what "servers" are in this email. Do servers mean each virtual server so it is going to take hundreds of hours from now or are "servers" each node and everything should be restored in a few hours?


----------



## zulualpha

leeboof said:


> I wonder what "servers" are in this email. Do servers mean each virtual server so it is going to take hundreds of hours from now or are "servers" each node and everything should be restored in a few hours?


Was wondering the same thing, but I am assuming they're talking about nodes...otherwise we could be looking at being back online sometime next month....


----------



## infinityhosting

I just want a simple answer to when I will be in control of my vps again.


----------



## drmike

*For those of you that are customers:*

If you send me a private message I can look your account up and determine which server you are on.  That will help going forward to know what the issue is and independently do some self determination on resolution and patience.

Customers, if they knew the node, could look at say Pingdom and see if they are a node that is entirely offline or publicly clear having problems today.

*Nodes with uptime issues today:*

buf-vps14 - 3h 28min downtime

chi-vps13 - 0% uptime

chi-vps14 14h 32min downtime

chi-vps16  - 0% uptime

chi-vps17 - 0% uptime

chi-vps23 - 0% uptime

chi-vps24 - 0% uptime

chi-vps32 - 0% uptime

chi-vps40 - 3h 37min downtime


----------



## Nth

infinityhosting said:


> 8230
> 
> 
> This is an additional update. We are making great progress in restoring servers and our current pace is on average about 2 servers per hour.
> 
> 
> We are continuing to work tirelessly to restore your VPS, and working through our ticket queue as well. We are going as fast as possible and hope to fully resolve everything. Thank you again for your patience.
> 
> 
> Regards
> 
> 
> The ChicagoVPS Team


Two servers per hour and we know from the DB dump they have around 100 servers, so sometime between two days and now.

buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?


----------



## Chankster

Nth said:


> buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?


All DNS entries are as follows <Three Letter Location>-vps<Number>.chicagovps.net.  IE. CHI59 is chi-vps59.chicagovps.net. and yours would be chi-vps66.chicagovps.net


----------



## mnsalem

Nth said:


> Two servers per hour and we know from the DB dump they have around 100 servers, so sometime between two days and now.
> 
> buffalooed, I know my node but can't find it on pingdom, chi-vps66. Can you give me its IP?


But then again, remember! not all of these 100 are down!


----------



## funzie

buffalooed said:


> *For those of you that are customers:*
> 
> If you send me a private message I can look your account up and determine which server you are on.  That will help going forward to know what the issue is and independently do some self determination on resolution and patience.
> 
> Customers, if they knew the node, could look at say Pingdom and see if they are a node that is entirely offline or publicly clear having problems today.
> 
> *Nodes with uptime issues today:*
> 
> buf-vps14 - 3h 28min downtime
> 
> chi-vps13 - 0% uptime
> 
> chi-vps14 14h 32min downtime
> 
> chi-vps16  - 0% uptime
> 
> chi-vps17 - 0% uptime
> 
> chi-vps23 - 0% uptime
> 
> chi-vps24 - 0% uptime
> 
> chi-vps32 - 0% uptime
> 
> chi-vps40 - 3h 37min downtime


I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.


----------



## drmike

Nth said:


> chi-vps66. Can you give me its IP?


 

IP looks like: 198.46.156.2


----------



## drmike

funzie said:


> I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.


ATL-VPS2 was one of the nodes that particularly hard hit and they didn't have any backups in Atlanta.

Technically the server there has been rebuilt and is up.  But the VPS containers for customers is GONE.


----------



## drmike

Someone that is a provider / familiar with Solus answer this:

In nodes table, there is ftpbackup value, a 0 or 1.   Assuming 0 means ftpbackups are not enabled and a 1 means ftpbackups for the node are enabled.

Does that include client containers in that form of backup?

I see 26 nodes where ftpbackup is set as 0.


----------



## upsetcvps

Another useless update.


----------



## jer

funzie said:


> I have been reading this thread since it started. I am posting now because I am on atl-vps2 which according to Pingdom has been up this whole time. But my VPS went down along with the others. I don't really believe the validity of those stats.


That's what I was trying to say with my first post, but better said.


----------



## drmike

mysql> select nodeid,name from nodes where ftpbackup = 0;

+--------+-----------+

| nodeid | name      |

|      1 | localhost |

|     35 | chi22     |

|     21 | chi10     |

|     25 | chi13     |

|     31 | chi18     |

|     37 | chi24     |

|     39 | chi23     |

|     42 | chi27     |

|     79 | chissd1   |

|     48 | chi32     |

|     49 | chi33     |

|     57 | chi40     |

|     65 | chi47     |

|     68 | chi50     |

|     76 | chi51     |

|     80 | chi53     |

|    109 | atl1      |

|    110 | atl2      |

|    128 | atl3      |

|    131 | atl4      |

|    133 | atl5      |

|    138 | atl6      |

|    148 | nj1       |

|    149 | dfw1      |

|    150 | njkvm1    |

|    151 | chi70     |

+--------+-----------+


26 rows in set (0.00 sec)

 



ATL-VPS2 / ATL2 is on that list.  Those are the nodes where ftpbackups were not configured or were turned off. FTPBackups are the weekly backups that should be running automatically.

Someone asked about "centralbackups" manually ran by customers.  A provider with SolusVM experience will need to clarify where the centralbackups go if the ftpbackups are disabled.   I assume there is another setting / location for those --- hoping there is for those sitting and wondering.


----------



## rds100

The central backups go to special central backup server(s). But central backups are not made automatically, the user must go to SolusVM and make his own centralbackup.

By the way just because there is no ftp backup doesn't mean the node doesn't have backups. There are other ways to make node backups too, besiedes the solusvm ftpbackups feature.


----------



## Nth

Chankster said:


> All DNS entries are as follows <Three Letter Location>-vps<Number>.chicagovps.net.  IE. CHI59 is chi-vps59.chicagovps.net. and yours would be chi-vps66.chicagovps.net


Didn't even think to look it up by dns, thanks!


----------



## drmike

Another gotcha for folks still down:

If you are on a node that had extended downtime (5 hours+) in the past 3 days, you are on a node they most likely had to rebuild.

If you look at Pingdom, you can find the nodes in there with RED X's.   Those machines might be online, but the VPSes on them likely are not:

http://stats.pingdom.com/jzrszp4wfu79

I say that, because just received a few customer lookups and the servers show fine uptime today but customers are still offline (since Monday).


----------



## leeboof

buffalooed said:


> Another gotcha for folks still down:
> 
> If you are on a node that had extended downtime (5 hours+) in the past 3 days, you are on a node they most likely had to rebuild.
> 
> If you look at Pingdom, you can find the nodes in there with RED X's.   Those machines might be online, but the VPSes on them likely are not:
> 
> http://stats.pingdom.com/jzrszp4wfu79
> 
> I say that, because just received a few customer lookups and the servers show fine uptime today but customers are still offline (since Monday).


Yeah my node looks online but VPS is still down as well. My VPS wasn't listed in the backup SQL file so not having much hope for a recovery...

Another thing that's weird is my server doesn't match the node I thought I was on from previous emails. The node it shows in the database I am now shows as never being down. 

Not sure which is correct.


----------



## zero

*@**XFS_Duke*,  Simple as that. Just give it time.

time is over dude.

CVPS Mistakes;

1) Backup 

2) Disastery Plan

3) Customer Service and Communication

This is it ...


----------



## Chankster

Received this update on one of my tickets.

Hello,

We're getting pretty close to having this issue fully resolved. All nodes, and all customer containers will be back online in the next 24-36 hours. Further mass-update emails will be sent later today with additional updates. Thank you for your patience and understanding -- we appreciate your business very much.

---------------
Matthew
Support Guru


----------



## zero

I do not believe work everyting on 24-36 hour.

This is take gas in customers ...

How about backups ?

Whats The accuracy of backups ?

How day or week ago this backups ?

Whats the deleted vps customers status ?

How can give backups on lost vps's ?

How many vps up and runing now ?

Everything is blur ...

And chicago vps continues to persist communication ...

This is it CVPS not a institutional firm.


----------



## zero

We must a play a game.

*If I CVPS Ceo what is my next movement ?  This is key question*

_*If I CVPS Ceo* : I hire a customer relationship person for communication ..._


----------



## Chankster

Another update I received:

Dear Customer,

We're getting pretty close to having this issue fully resolved. All nodes, and all customer containers will be back online in the next 24-36 hours. Some containers will have all data intact and others will be fresh installs. For those with fresh installs, we may be able to restore your data from our backups, and if you used Central Backup your information will be available through that facility as well. Further mass-update emails will be sent later today with additional updates. Thank you for your patience and understanding -- we appreciate your business very much.

Josh
Support Guru


----------



## Francisco

zero said:


> We must a play a game.
> 
> *If I CVPS Ceo what is my next movement ?  This is key question*
> 
> _*If I CVPS Ceo* : I hire a customer relationship person for communication ..._


Isn't that why Adam was hired?

Francisco


----------



## leeboof

Chankster said:


> Another update I received:
> 
> Dear Customer,
> 
> 
> We're getting pretty close to having this issue fully resolved. All nodes, and all customer containers will be back online in the next 24-36 hours. Some containers will have all data intact and others will be fresh installs. For those with fresh installs, we may be able to restore your data from our backups, and if you used Central Backup your information will be available through that facility as well. Further mass-update emails will be sent later today with additional updates. Thank you for your patience and understanding -- we appreciate your business very much.
> 
> 
> Josh
> 
> 
> Support Guru


I got the same, they must be copy pasting into all open tickets..


----------



## HalfEatenPie

Well, they probably do have tons and tons of the same tickets...


----------



## concerto49

Francisco said:


> Isn't that why Adam was hired?
> 
> 
> Francisco


Kevin!


----------



## MannDude

I'm actually curious if he'll change his name in the backend or continue going by the fake name of Kevin Hillstrand. I don't expect the average customer to care who isn't aware of the Adam/Kevin issue.

Customers of CVPS: Has a fellow named Kevin Hillstrand responded to your tickets within the past few days? What about Adam Ng?


----------



## leeboof

MannDude said:


> I'm actually curious if he'll change his name in the backend or continue going by the fake name of Kevin Hillstrand. I don't expect the average customer to care who isn't aware of the Adam/Kevin issue.
> 
> Customers of CVPS: Has a fellow named Kevin Hillstrand *responded to your tickets* within the past few days? What about Adam Ng?


lol.. 

Only response the last few days has been the copy pasted message from above.


----------



## Nth

I've had Adam, Luc, Chris, and Matthew answer tickets.


----------



## codewarrior

Chankster said:


> Another update I received:
> 
> Dear Customer,
> 
> 
> We're getting pretty close to having this issue fully resolved. All nodes, and all customer containers will be back online in the next 24-36 hours. Some containers will have all data intact and others will be fresh installs. For those with fresh installs, we may be able to restore your data from our backups, and if you used Central Backup your information will be available through that facility as well. Further mass-update emails will be sent later today with additional updates. Thank you for your patience and understanding -- we appreciate your business very much.
> 
> 
> Josh
> 
> 
> Support Guru


Got same generic BS answer, after asking to PLEASE not give another generic BS answer.

Epic fail.

Colossal fail.

Need a new term for this place.


----------



## shawn_ky

Adam, Luc and Chris..


----------



## MannDude

Nth said:


> I've had Adam, Luc, Chris, and Matthew answer tickets.


Matthew is likely an alias. In the DB leak 'Matthew B' uses the email address [email protected] which is used for them to register domain names. That's probably Chris, if I were to guess. I've had him use that email address when contacting me in the past about an account ChicagoVPS had with the company I used to work for that was in the name of an old employee. We got it sorted out and changed to be in his name, and that was the email address used.

Interesting to see Adam in the mix. This is a kid who just turned 18 a month ago, who for a couple years was lying about his age, location and name going by 'Kevin Hillstrand'. The leak kind of outed him, despite a lot of us knowing for a long time (Him and the CVPS crew would deny deny deny). By chance was Adam posting as Adam N. or Adam Ng? (Or Adam Jackson, which he also goes by?). Just trying to figure out if this Adam fellow is the old Kevin, or if it's an entirely different person.


----------



## xvtv

Josh / Chris / Thomas / Kevin (mid-may)


----------



## srichter

So they say it's too much work to restore everyone's backup and to open a ticket if you want your backup restored. Then they reply to that ticket with a generic email saying they're restoring anyone anyway and your data might be there or might not...


----------



## Nth

He signed it Adam Ng. On another interesting note; my canned responce letter like chankster's above was signed Matthew not Josh. Wonder if they got colocrossing people copy/pasting that to everyone with an open ticket or if they have a script with amnesia.


----------



## MannDude

xvtv said:


> Josh / Chris / Thomas / Kevin (mid-may)


Was the Kevin response today?



Nth said:


> He signed it Adam Ng. On another interesting note; my canned responce letter like chankster's above was signed Matthew not Josh. Wonder if they got colocrossing people copy/pasting that to everyone with an open ticket or if they have a script with amnesia.


I'm 88% sure Matthew is just another alias, likely used by Chris. The other names may be for CC staff as they _are_ helping CVPS to restore things back to normal but don't want to appear to be involved in doing so. (Although we already know this as other paying customers of Colocrossing had things delayed because their staff was busy working on restoring CVPS customer's data).

Though it is interesting that Adam is finally using his real name. That's a step in the right direction.


----------



## Francisco

MannDude said:


> I'm 88% sure Matthew is just another alias, likely used by Chris.


Easy way to prove who it is. Pull the adminlogs table of the dbdump for the IP used for actions. There's a slew of CC IP's in the WHT thread to compare to.

Francisco


----------



## leeboof

Latest Update:



> I just wanted to update everyone and let them know that things are starting to calm down and return to normal. Communication moving forward will be much better since it is now less hectic and we appreciate everyone’s patience as it has been a long 3 days. Ticket response times will be much better and the pace has really picked up as we are concluding our recovery.
> 
> We appreciate your support and patience!


----------



## MannDude

Francisco said:


> Easy way to prove who it is. Pull the adminlogs table of the dbdump for the IP used for actions. There's a slew of CC IP's in the WHT thread to compare to.
> 
> 
> Francisco


Too much effort. Matthew B., the other admin in the DB and I assume the same 'Matthew' answering tickets is Chris Fabozzi. I say this because 'Matthew B.' was using the email [email protected] and I doubt the owner of the company would share his email address with an employee.


----------



## concerto49

Why would Chris need an alias but?


----------



## jer

I had asked about the node being up in pingdom but the vps down. Here's the answer I got, that may help others:



> Jer,
> 
> Some nodes are online but the containers are damaged. We will have all that sorted in the next 24 hours. We are working extra hard to get this done ASAP.
> 
> Thank you for your patience.
> 
> ---------------
> Matthew


----------



## drmike

leeboof said:


> Another thing that's weird is my server doesn't match the node I thought I was on from previous emails. The node it shows in the database I am now shows as never being down.


CVPS has a long and soiled history about moving people around after you are a customer.  It happens most often when you complain about performance of the server you are on and this move you to one with better neighbors.  There are other reasons to have been moved - like a prior server failure.

Send me a private message and I'll double check your node info.


----------



## drmike

MannDude said:


> Though it is interesting that Adam is finally using his real name. That's a step in the right direction.


 

Well, Adam answered at least on ticket as himself back in December 





 







> Boom Scape · 7,963 like this
> 
> December 9, 2012 at 7:12am ·
> 
> 
> 
> 
> 
> 
> 
> 
> Just received some good news guys
> 
> "James,
> 
> We are installing Windows on your VPS now.
> 
> ---------------
> Adam Ng
> ChicagoVPS Support Tech"







 




https://www.facebook.com/permalink.php?id=204958342858533&story_fbid=508822079138823

Fact is, that was one of the pieces of info that implicated Adam as being Kevin way back then


----------



## drmike

concerto49 said:


> Why would Chris need an alias but?


Well the anon email has existed for eons (years?)

Hard to say why Chris created it and who all might be portaling through it to be whoever. 

I can tell you who is absent from the admin list directly --- Colocrossing employees.

This is the admin list from November hack:

mysql> select adminid, username, emailaddress  from administrators;

+---------+----------+---------------------------+

| adminid | username | emailaddress              |

+---------+----------+---------------------------+

|       1 | vpsadmin | [email protected]      |

|       2 | jshinkle | [email protected]   |

|       4 | layotte  | [email protected]    |

|       8 | jsantos  | [email protected]    |

|      10 | lgibbons | [email protected] |

+---------+----------+---------------------------+

 

 

This is the current admin list:


select adminid, username, emailaddress  from administrators;

+---------+----------+-------------------------+

| adminid | username | emailaddress            |

+---------+----------+-------------------------+

|       1 | vpsadmin | [email protected]    |

|       4 | layotte  | [email protected]  |

|      16 | fabocj40 | [email protected]    |

|      15 | tleonard | [email protected] |

|      12 | adamng   | [email protected]      |

|      14 | matthew  | [email protected]     |

+---------+----------+-------------------------+

 


Gone are Shinkle, Santos and Gibbons.

New are fabocj40, tleonard, adamng and matthew.

Fabocj40 is the company owner at CVPS as is the matthew account.

I'll go dig into the IP stuff next


----------



## MannDude

He's been unbanned from LET so I guess he'll unlikely grace us with his presence: http://www.lowendtalk.com/discussion/11304/chicagovps-update


----------



## drmike

MannDude said:


> He's been unbanned from LET


 

He was never "banned".  Chris said on here that he asked LET to ban him as the site was consuming too much of his time.  A self ban.


----------



## xvtv

> I just wanted to update everyone and let them know that things are starting to calm down and return to normal. Communication moving forward will be much better since it is now less hectic and we appreciate everyone’s patience as it has been a long 3 days. Ticket response times will be much better and the pace has really picked up as we are concluding our recovery.
> 
> 
> We appreciate your support and patience!
> 
> 
> Thank you,
> 
> 
> The ChicagoVPS Team


Still waiting for mine to be up...

edit: Last support ticket from Kevin was ~ one month ago...


----------



## shawn_ky

Anyone see anything about their Dallas location? Haven't seen anything that I remember...


----------



## drmike

Dallas = 1 server:

 149 | dfw1    |        5 |         5.75 |

That is node 149.

5 VPS accounts with 5.75GB sold RAM.

It is the second lowest RAM allocation and is a well undersold node.


----------



## srichter

Over @ LET:



> Further good progress has been made and we are down to the final list of machines that were affected. More than 50% that were affected have been fully restored with files intact and the remaining list that needs more attention is what is left and we are working very hard to restore them fully with files intact.
> 
> Thanks for all your patience!


----------



## xvtv

50% in 3 days... I don't know if that's really good news!


----------



## drmike

srichter said:


> . More than 50% that were affected have been fully restored with files intact



affected vs. effected ?

50% success rate?  Ho hum.  Listen to the customers moan.


----------



## drmike

One more thing to note, Pingdom DOES NOT have all nodes under monitoring.

Dallas isn't in there and those 5 VPSes are offline.

New Jersey is also absent from monitoring and status of there is unknown.


----------



## mpkossen

buffalooed said:


> He was never "banned". Chris said on here that he asked LET to ban him as the site was consuming too much of his time. A self ban.


Oh, he was banned. By Chief and not on his own request.

Not that he doesn't have a second account on LET, but still.


----------



## drmike

mpkossen said:


> Oh, he was banned. By Chief and not on his own request.


 

Oh man, another Fabozzi lie then 

@mpkossen, they are lucky to have you help over on the low end.  Glad to see you on vpsBoard!


----------



## srichter

> Just a quick update to keep everyone in the loop. Within the next 6 hours we will have all servers reinstalled ready to restore backups on the remaining machines. Within 18 hours we expect to resume normal status and all customers to have running VPS' with data intact.
> 
> 
> Thank you again for your patience!
> 
> 
> Regards
> 
> 
> The ChicagoVPS Team


----------



## mikho

mpkossen said:


> Oh, he was banned. By Chief and not on his own request.
> 
> 
> Not that he doesn't have a second account on LET, but still.


Well, Chris was banned by Chief more then once and then had the ban lifted.


Liam once said that you could see who banned an account? Then that would be the last thing Joel did before leaving the community (after the sale was finished)


----------



## mpkossen

mikho said:


> Well, Chris was banned by Chief more then once and then had the ban lifted.
> 
> 
> Liam once said that you could see who banned an account? Then that would be the last thing Joel did before leaving the community (after the sale was finished)


Probably, yeah.

All these politics ;-)


----------



## walesmd

So, wait... this vulnerability was announced before and and CVPS did nothing to thwart it? Thank goodness I don't do anything serious on my VPS there - just couldn't pass on the deal - didn't even notice I was down until 24 hours or so afterwards, when they were sending emails.


They've tried to maintain a line of communication but it doesn't really make any sense - just different ways of saying "we're working on restoring, you're screwed until we say otherwise". Some emails it sounded like everyone was getting restored, others made it seem like we'd all go blank and could submit a ticket for a restore.


----------



## Lanarchy

CHI is up, NY is up, LA is up and restored with original data

ATL, 1 is available and appears wiped + fresh image, even though I have no idea what the root password is. And one is up, but totally inaccessible and I think it has the data on it, but the firewall is haywire or something and I cannot reboot it with no control panel.


----------



## walesmd

ATL here as well, can't even ping.


----------



## jfreak53

> Just a quick update to keep everyone in the loop. Within the next 6 hours we will have all servers reinstalled ready to restore backups on the remaining machines. Within 18 hours we expect to resume normal status and all customers to have running VPS' with data intact.
> Thank you again for your patience!
> 
> Regards
> 
> The ChicagoVPS Team


So that email was sent last night at around midnight, 11:58PM to be exact. It says 6 hours all will be restored and then backups, but yet on Pingdom there are still 5 Chicago nodes offline ha ha







Wouldn't that have been like an hour ago ha ha? I might expect to see 1 offline still, but not the same 5.


----------



## upsetcvps

Well I'm glad they are at least giving time-frames now.  It's ok if they're a little off.  I wish I knew what exactly they are doing though.  Did they expect to have everyone up with a fresh vps in 6 hours (and how do I log in?) and then have the backups restored 12 hours after that?

Anyway, this is a slight improvement in communication.  Hopefully by the end of the day they will have everything sorted out.


----------



## ramas

upsetcvps said:


> Well I'm glad they are at least giving time-frames now.


As long as they're not giving random numbers just to calm their clients down. 2 days ago they said: "[SIZE=small]Our goal is to have everything 100% restored tomorrow."...[/SIZE]



upsetcvps said:


> Anyway, this is a slight improvement in communication.


Agreed.

They also mentioned they're dropping SolusVM completely and will look for an alternative. I'm wondering how long that's going to take and how're we suppose to control our VPS (once it's up and running) in a meantime.


----------



## cvps_customer

Learned my lesson on cheap vps's, will def hop over to linode. The responses i've gotten from [email protected] have been comical and contradict the emails that were sent out to all users. 

Corrupted backups? Testing your backups is Sys Admin 101, do these guys have any idea what they are doing? Going on 4 days downtime now, very glad I'm not relying on their service to host any sites of my own. If they didn't oversell their boxes they would easily be able to provide customers a VPS to use on one of the working nodes while working to restore data on the failed ones.

*EDIT* Looking back through the thread I noticed buffalo's post regarding ftp backups being disabled. This is absolutely ridiculous, such a joke of a business.


----------



## upsetcvps

cvps_customer said:


> Learned my lesson on cheap vps's, will def hop over to linode. The responses i've gotten from [email protected] have been comical and contradict the emails that were sent out to all users.
> 
> 
> Corrupted backups? Testing your backups is Sys Admin 101, do these guys have any idea what they are doing? Going on 4 days downtime now, very glad I'm not relying on their service to host any sites of my own. If they didn't oversell their boxes they would easily be able to provide customers a VPS to use on one of the working nodes while working to restore data on the failed ones.
> 
> *EDIT* Looking back through the thread I noticed buffalo's post regarding ftp backups being disabled. This is absolutely ridiculous, such a joke of a business.


That just means they weren't use the solusvm backup solution there.  They could have had their own.  Now they have definitely said some nodes don't have backups or were corrupted but not having ftpbackup through solusvm in particular isn't that terrible


----------



## cvps_customer

upsetcvps said:


> That just means they weren't use the solusvm backup solution there.  They could have had their own.  Now they have definitely said some nodes don't have backups or were corrupted but not having ftpbackup through solusvm in particular isn't that terrible


Thanks for the clarification, still hard to believe they don't have some type disaster recovery plan in place.


----------



## jer

Just wanted to note that I'm on CHI-VPS10 - which reports up in pingdom, however my vps is not. I was told the reason for this is because some containers are damaged.

I have some 'container' experience. Both Virtualization Professional certs and the Virtualization Engineer cert from Parallels (not solusvm).

After looking at the CVPS dump for me and my container, I can say that the parameters for my container appear to be intact, and fine.

My node was done yesterday morning, I didn't ask for a restore, my container parameters don't appear damaged, and I'm still down.

I've had a long time suspicion that Chris leaves people down if he doesn't like them, and purposefully toys with them. I've confronted him before on it. I think this is one of those cases.

(continued)


----------



## jer

(Mods: I made this it's own post so it can be removed if needed without loosing the info in the first post.)

I've seen Chris call his customers "Idiots" on LET, had a long time suspicion that some employees were personas (now confirmed), Shinkle "terminated" my account (dropped my vps) for non-payment - on the day the payment was due (not the day after) - while I had a 2 day old open support ticket about how I couldn't get their payment portal to take my payment, that Shinkle had responded in. That was a while back. It's a good thing I know to keep good backups, imagine the impact to a regular Joe. Add that downtime to the previous Outage + this one.

Last month their payment system charged me twice. Chris gave me a credit (and avoided a charge back). So I'm 'pre-paid' for next month. It's one of the reasons I didn't switch providers 4 days ago.

There's an entire list, with tickets and conversations I've archived as far as 2 years back.

I put up with it because my sites aren't commercial, the network and throughput are good, and I'm getting a great deal.

I don't blame Chris for the SolusVM incident. I do blame Chris for how he's handled it. That does tack on to a growing list of how he's handled previous issues, and creates an opinion that Chris is not mature enough to run a company.

----------

If anyone can offer the same or nearly the same deal (or knows of a similar deal), is not a reseller of CVPS service and is not associated with a company Chris has had his hands in - I'll defect.

I currently have 2 IPv4s, 512M, 20G, Xen for $5.95. Network and throughput are important and prefer the host server located somewhere in the middle of the US. Please let me know via PM, not in the thread.


----------



## upsetcvps

jer said:


> I currently have 2 IPv4s, 512M, 20G, Zen for $5.95. Network and throughput are important and prefer the host server located somewhere in the middle of the US. Please let me know via PM, not in the thread.


I'm interested in alternative offers too.  Please post in thread!


----------



## ramas

I'm annoyed as much as you are. I have a numerous projects I'm working on and obviously I have my own servers/environments/etc that I manage. Even though I don't _really_ need a VPS, I figured heck if someone can take care of all the hardware/networking stuff for me and just give me a decent-usable environment for such a minimal price - I simply couldn't resist an offer and moved testing of some of my projects to ChicagoVPS.
Bet in general I really don't want to jump on this whole "bashing ChicagoVPS and even going personal" bandwagon. Sh*t happens I guess.

Now regarding the backups and some people claiming it's "clients fault" - I just don't agree. First of all, normally I wouldn't care less about the backups and such since that's not why *I* got my VPS for, EXCEPT for the fact that they list it as a service they provide. And it's not just some random advertisement wording strategy (which I would be fine with - everyone does it), but they have actually written it straight in their end-user agreement that I "signed" (stating: "fail-over protection and backup systems.").
In other words I'd have still got their VPS even if they clearly indicated they have no backups nor are responsible for my data; but since they agreed to provide me such service - obviously I'm quite disappointed.
For example "GreenValueHost" (from the top of my head) says they provide SSL certificates for their VPS. Great, and if I wouldn't get one after signing in I would be rightfully annoyed, no? Same with backups - if they say they have my backup - I expect them to have my backup (and there's a difference between "hackers damaged our backups" vs "we have no backups for some nodes").



jer said:


> If anyone can offer the same or nearly the same deal (or knows of a similar deal), is not a reseller of CVPS service and is not associated with a company Chris has had his hands in - I'll defect.


Unfortunately, I believe that's the point. As an old saying goes - you get what you paid for. If there was an alternative that has the same price and is "better", 99% of people would move there.

On a bright side with all this happening at least I learned what are the major aspects of a decent VPS/dedicated-server provider


----------



## jer

That's what I used my VPS for - a project dev environment. They're coming of age (the projects), but I don't make income from them so I don't want to go "full Linode" (although their service is outstanding IMO).

I don't think I can get a better price, the deal I have is great. I do think I can get a similar price for the same service.


----------



## Amitz

Look at http://Overzold.com (Prometeus) and http://www.waveride.at (Edis).


----------



## Chankster

So RFO #7 from 9hrs ago stated that all VPS's would be reinstalled and ready to restore backups in 6hrs.  Update I just got from a ticket stated that its going to be 12 hours before they hit that point.
 



> We are getting closer to having all services restored -- at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time.
> Once we've reached that point we will begin restoring customer containers for those who have requested it.
> Doing so is a highly manual process so it will take quite some time; if you have your own backups you will be able to restore your data faster than we can.
> 
> Additionally we plan to re-enable SolusVM shortly as we are in the final stages of our security audit.
> 
> Thank you again for your patience as we work to get everything back online and running.
> 
> Josh
> Support Guru


----------



## leeboof

Chankster said:


> So RFO #7 from 9hrs ago stated that all VPS's would be reinstalled and ready to restore backups in 6hrs.  Update I just got from a ticket stated that its going to be 12 hours before they hit that point.


12 hours from now or does that mean 3 more hours supposedly?


----------



## Chankster

leeboof said:


> 12 hours from now or does that mean 3 more hours supposedly?


Sounds like 12 hours from now.  From ticket: "...at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time."


----------



## mnsalem

Chankster said:


> Sounds like 12 hours from now.  From ticket: "...at this point we expect all nodes to be online within 12 hours and all customer containers to be online and available at that time."


Most nodes are indeed back up according to Pingdom ... only 4 Chicago nodes left! 


If they're still working on the 2 servers per hour rate ... then i think the nodes will be ready in the next couple of hours for operation!


----------



## Chankster

mnsalem said:


> Most nodes are indeed back up according to Pingdom ... only 4 Chicago nodes left!
> 
> 
> If they're still working on the 2 servers per hour rate ... then i think the nodes will be ready in the next couple of hours for operation!


Remember that nodes being up does not equal customer containers being up.  Pingdom shows that CHI59 has only had 3.75 hours of total down time but my container has been down since the morning of the 18th.


----------



## upsetcvps

I'm curious: has anyone who was down gotten there node back up since then?  With data or fresh?


----------



## mnsalem

Chankster said:


> Remember that nodes being up does not equal customer containers being up.  Pingdom shows that CHI59 has only had 3.75 hours of total down time but my container has been down since the morning of the 18th.


Ofcourse ... don't forget they need to get the host NODES up first THEN work on the CONTAINERS 


But it is getting closer finally


----------



## jfreak53

upsetcvps said:


> I'm curious: has anyone who was down gotten there node back up since then?  With data or fresh?


One single server of the 10 of mine that are offline has come back up, albeit an empty install and I have no clue on God's green earth what the root password is to login either. I requested this server be restored from backup yesterday morning, so it's up but still no backup restored. Nor can I login to restore my own backups since I don't know what the root pass is haha.

About every hour they respond to another of my 20 tickets that are currently open pasting that same response in. What's funny is two hours ago that pasted response said they would have Solus back online shortly, well that's 2 hours ago ha ha.


----------



## jacobsta811

Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.


----------



## zulualpha

jacobsta811 said:


> Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.


So they're not sending out new root passwords after the DB leak? It's a good thing we live in such an honest world.  :unsure:

Was there any notification to you when your VPS came back up, or do we just have to keep trying indefinitely?


----------



## upsetcvps

jfreak53 said:


> One single server of the 10 of mine that are offline has come back up, albeit an empty install and I have no clue on God's green earth what the root password is to login either. I requested this server be restored from backup yesterday morning, so it's up but still no backup restored. Nor can I login to restore my own backups since I don't know what the root pass is haha.
> 
> About every hour they respond to another of my 20 tickets that are currently open pasting that same response in. What's funny is two hours ago that pasted response said they would have Solus back online shortly, well that's 2 hours ago ha ha.


They're bringing solus back up?  I think it's stupid for a provider like this to run software without access to the source.  Chris said no one listened to him last time and that's the reason this happened again.  So his solution is what?  Hope things don't happen in threes?


----------



## jacobsta811

I haven't gotten any new root passwords, but I agree they really should be resetting root passwords.

Edit: they have to bring back up a control panel sometime. Given their resources it seems to me it would be easier to just fix solus and the containers and use the labor of the customers to reimage, reset root passwords when they do, etc. Rather than try to do it themselves. Unfortunately that opens up the risk of another attack, but running with any decent number of customers basically requires the customers can do it themself.


----------



## jfreak53

jacobsta811 said:


> Of 4 (Atlanta, Chicago, Buffalo, LA), Atlanta came back up first, was totally wiped, and there was some kind of issue so I just changed the password and shut it down. Chicago came back up yesterday and appeared to be intact (although I had hardly anything in there as it wasn't in use yet- just my standard setup where I setup automatic updating, block everything with IPTables and install logwatch and fail2ban). I changed the password and left it up. Chicago is now *down* again though, not sure why, and obviously, no way to boot it. My root passwords were the original (presumably exposed) ones, but yours could have been changed by a hacker before you saw it was up, since there are not emails going out the second your node is restored or imaged.


The server in question that came up has no password available for login, it is only able to be logged in by someone using either serial console or SSH Auth, no password. So not possible considering SolusVM isn't up yet. Which means it's a fresh install and no backup placed yet.

Second, I never used nor do I EVER use Solus to change root password, I only ever use 'passwd' from root prompt, meaning solus has no clue what my root password was for any of my servers


----------



## jfreak53

upsetcvps said:


> They're bringing solus back up?  I think it's stupid for a provider like this to run software without access to the source.  Chris said no one listened to him last time and that's the reason this happened again.  So his solution is what?  Hope things don't happen in threes?


Of course, Solus finished their Security audit yesterday around 5 I think. Most providers brought solus online last night or early this morning. So as long as they have installed the most recent version patched they should be good. At least that's what I've been told by other providers.


----------



## MannDude

Woah, wait? They're not forcing a password reset on all customers? You've surely kidding. Well, if that's the case, yeah... change your password immediately when you get your VPS back. Who knows how many people has that DB leak now. It's bad enough your email and name is in it, even worse a password that you may or may not use in other places is in it too.

Change your password for anything that shares that.

Back in November when the DB was leaked the 1st time, Chris Fabozzi's password was in there for the admin login. He used the same password in other places, including Skype and LET, etc. Someone took his password from the DB leak, and had a bit of fun I believe. He should know the importance of informing his own customers to change their PWs.


----------



## upsetcvps

MannDude said:


> Woah, wait? They're not forcing a password reset on all customers? You've surely kidding. Well, if that's the case, yeah... change your password immediately when you get your VPS back. Who knows how many people has that DB leak now. It's bad enough your email and name is in it, even worse a password that you may or may not use in other places is in it too.
> 
> Change your password for anything that shares that.
> 
> Back in November when the DB was leaked the 1st time, Chris Fabozzi's password was in there for the admin login. He used the same password in other places, including Skype and LET, etc. Someone took his password from the DB leak, and had a bit of fun I believe. He should know the importance of informing his own customers to change their PWs.


 


The db also does not have my root password as I never used solusvm.  But I will be doing a fresh install anyway and restoring data from my own safe backups.

 

This is also why I use fake data for these cheap boxes.  Fake (well, not fake, but made just for the vps) e-mail, fake address, fake name.  The money is real; My data, well you cannot protect it apparently, so why should you have it.


----------



## srichter

Still can't ping my VPS on buf-17


----------



## jfreak53

I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.


----------



## upsetcvps

jfreak53 said:


> I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.


They're probably all on a plane to Brazil...


----------



## SkylarM

jfreak53 said:


> I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.


Hopefully for the sake of the clients, that isn't the case.

It's pretty interesting seeing something like this happen twice, and see how two totally different companies handle similar issues. Huge hats off to RamNode for dealing with it properly I must say.


----------



## maounique

In all honesty ramnode had much fewer nodes.

However, it is true CVPS could have handled it much better, especially since they already had this experience once.


----------



## mikho

Mao said:


> In all honesty ramnode had much fewer nodes.


And not to forget, less vps / node.


----------



## mnsalem

Latest Report in right now:



> Hey everyone, just a quick update. Since our last email, all nodes that were still affected have since been reinstalled. Right now we are working on installing the new VPS for each server and we are still making good progress.
> 
> 
> When we reach final completion we will release another update.
> 
> Thank you all for your patience.
> 
> Regards
> 
> The ChicagoVPS Team


----------



## maounique

mikho said:


> And not to forget, less vps / node.


That you cant know, could have been larger nodes, for example, in the end, it matters mostly the quantity of data, that is where the most delay should be.

Since Ramnode uses ssd on many nodes, i can figure the storage space is not that big, therefore restores should be faster.

But, again, even so, it took horibly long time to restore for CVPS. We are closing to a week now. Could have been faster, I think, but I do not have all the data so i could be wrong.


----------



## jfreak53

They say everything is back online but Pingdom still shows 4 Nodes down ha ha.


----------



## mnsalem

jfreak53 said:


> They say everything is back online but Pingdom still shows 4 Nodes down ha ha.


ya 


But then you suspect if these were ever online! like, were they even online before the issues that occured? maybe they never were 


Unless someone can confirm they were active.


----------



## upsetcvps

> Hey everyone, just a quick update. Since our last email, all nodes that were still affected have since been reinstalled. Right now we are working on installing the new VPS for each server and we are still making good progress.
> 
> 
> When we reach final completion we will release another update.
> 
> 
> Thank you all for your patience.
> 
> Regards
> 
> 
> The ChicagoVPS Team



I don't know what this means.  The last e-mail suggested that 2 hours from now everything would be restored.  Is that still the case...?  Sigh.


----------



## Aldryic C'boas

> Since our last email, all nodes that were still affected have since been reinstalled.


This reads a bit like _"if you don't have your own backups on these nodes, you're probably SOL"_.


----------



## drmike

Mao said:


> In all honesty ramnode had much fewer nodes.


Well, I've long be curious as to number of nodes RamNode operates.   I don't think they are as small as folks think. Seem to have quite a few customers.  Ahh where's the SolusVM database  ?

What RamNode really had was a better management policy and contingency planning.  Nick isn't a sales idiot, he's the owner and a technical guy.  Contrast that to CVPS where the owner is a sales fellow who spends his time trolling forums and playing whack-a-mole.

Then again, no clue in either attack to the number of nodes destroyed.  I suspect CVPS had far more nodes destroyed.



jfreak53 said:


> washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response



Little doubt at this point if you are offline, you are going to receive an empty VPS --- if you still want to be a CVPS customer. (heck two major hacks in 7 months --- so the saying goes, third one is a charm).

The canned responses are kind of comical.   This is why you hire proper staff and maintain what you need for the customer base.   They should have brought in more folks to do customer support and someone to get working on billing credits while the other helpers and regular staff dealt with the technical issues of reinstalling servers and retrieving backups.

Those offline have been the entire business week.



jfreak53 said:


> They say everything is back online


There are at *LEAST* 11 servers that show major downtime in this 24 hours.  Mind you, there are nodes that aren't monitored in Pingdom and one of them had been down through at least early this morning.  Plus their control panel is offline according to monitoring (unsure how customers are doing Solus-necessary things without Solus available).


----------



## Marc M.

opcorn:


----------



## Nth

buffalooed said:


> Plus their control panel is offline according to monitoring (unsure how customers are doing Solus-necessary things without Solus available).


We aren't. My VPS has been online for a few hours but its just a fresh install as they said. I logged in just long enough to see that and logged out. I'll wait a couple more days to see if they can restore it before settings everything back up. It is good that they're making progress.


----------



## upsetcvps

Nth said:


> We aren't. My VPS has been online for a few hours but its just a fresh install as they said. I logged in just long enough to see that and logged out. I'll wait a couple more days to see if they can restore it before settings everything back up. It is good that they're making progress.


How did you obtain the login credentials?


----------



## XFS_Duke

jfreak53 said:


> I think they've just washed their hands of it and are just waiting for people to jump ship personally. Each ticket opened gets the same canned response. Same 4 servers offline as when this whole thing started. From a "Josh" someone ha ha.


I saw them restore your VPS if I'm not mistaken, your VPS should be ready soon.


----------



## infinityhosting

XFS_Duke said:


> I saw them restore your VPS if I'm not mistaken, your VPS should be ready soon.


Could I pm you my information and could you check on the status of my vps?


----------



## jfreak53

XFS_Duke said:


> I saw them restore your VPS if I'm not mistaken, your VPS should be ready soon.


Which one?  :mellow:


----------



## Nth

upsetcvps said:


> How did you obtain the login credentials?


It is the orginal password emailed to you when you first got your VPS.


----------



## zulualpha

Nth said:


> It is the orginal password emailed to you when you first got your VPS.


Are they imaging the VPS with the distro you selected when you first got your VPS or the one you were using when everything went down?


----------



## XFS_Duke

zulualpha said:


> Are they imaging the VPS with the distro you selected when you first got your VPS or the one you were using when everything went down?


Should be the one that you were using when it went down...


----------



## Nth

zulualpha said:


> Are they imaging the VPS with the distro you selected when you first got your VPS or the one you were using when everything went down?


Can't say as I never switched distros.


----------



## MannDude

XFS_Duke said:


> Should be the one that you were using when it went down...


Since you're helping CVPS restore customer's VPSes, can you comment on why they didn't do a forced password change?

In March when the November DB was leaked, they forced a reset on all passwords as a 'security measure' (later to be revealed it was because the DB was leaked), so I know they know how to.

Just seems a bit silly re-creating these VPSes with the same credentials. Judging by the number of search queries I've seen for people looking for the DB as well as people asking on Twitter for the DB, I'd be quite alarmed by who all has hands on it now. I'm sure your average customer should know to change their password immediately, even better using the 'passwd' command via SSH, but still...


----------



## zulualpha

XFS_Duke said:


> Should be the one that you were using when it went down...


Well that will make life a little easier... Are they emailing people when their VPS goes back up or should we just keep trying to ssh until it eventually works?


----------



## upsetcvps

Nth said:


> It is the orginal password emailed to you when you first got your VPS.


what a stupid policy.  How are they sure whoever hacked them didn't obtain access to these?  Why don't they just generate new passwords and send out an e-mail like gets done when you first create an account?


----------



## Nth

zulualpha said:


> Well that will make life a little easier... Are they emailing people when their VPS goes back up or should we just keep trying to ssh until it eventually works?


I got no message. Since it was down I've been running ping with a 30 second timeout on my vps's ip and noticed when it came back up.


----------



## xvtv

My vps is finally up after 87 hours !

But I can't login. My ssh key is not recognized anymore, and the original password is not working as well.

Have to wait for solusvm...


----------



## earl

It's times like these that makes you wonder about those companies that ask for government issued ID's for a $1 VPS!

Hmm.. no thanks! lol


----------



## Tactical

LOL!


----------



## mnsalem

VPS finally up here .. With the wrong OS 


When I ordered, I chose Debian, now from the apache test page, I see it's CentOS


Waiting for the backup restoration now!


----------



## srichter

mnsalem said:


> VPS finally up here .. With the wrong OS
> 
> 
> When I ordered, I chose Debian, now from the apache test page, I see it's CentOS
> 
> 
> Waiting for the backup restoration now!


What location/node are you on?


----------



## mnsalem

BUF19 as far as I can remember


----------



## srichter

mnsalem said:


> BUF19 as far as I can remember


I'm on buf17 and still down. Guess they're not going in order (or my node was trashed worse).


----------



## drmike

Well we are at the 5 day mark.  120 hours now downtime for folks still offline ...

Who else is still down and what server/location?


----------



## srichter

> As we finish up installing the new VPS' on the final machines I wanted to give an update. Within the next 5 hours, all OpenVZ VPS' should be installed and completed ready for you to restore.
> 
> We still have a few Xen machines that had severe damage to them, we are still going to working on them and will be ready tomorrow. I will make a decision tonight about compesation and release another email in the morning. We would like to thank everyone once again for all your patience.
> 
> Here at ChicagoVPS we want to ensure this will never happen again in the future. We are in the progress of working closely with a security firm for a complete audit over our infrastructure.
> 
> Thank you
> 
> The ChicagoVPS Team


----------



## mnsalem

srichter said:


> As we finish up installing the new VPS' on the final machines I wanted to give an update. Within the next 5 hours, all OpenVZ VPS' should be installed and completed ready for you to restore.
> 
> We still have a few Xen machines that had severe damage to them, we are still going to working on them and will be ready tomorrow. I will make a decision tonight about compesation and release another email in the morning. We would like to thank everyone once again for all your patience.
> 
> Here at ChicagoVPS we want to ensure this will never happen again in the future. We are in the progress of working closely with a security firm for a complete audit over our infrastructure.
> 
> Thank you
> 
> The ChicagoVPS Team
Click to expand...

"completed ready for you to restore" ???


So .. no news about the backups at alll? :/


----------



## srichter

srichter said:


> We are in the progress of working closely with a security firm for a complete audit over our infrastructure.
Click to expand...

Who wants to bet they mean SolusVM's audit? Also, you'd think the security firm would have told them not to restore using the leaked root passwords.


----------



## drmike

srichter said:


> srichter, on 22 Jun 2013 - 04:07 AM, said:   Quote We are in the progress of working closely with a security firm for a complete audit over our infrastructure.


 

Oh boy.   That quotable he'll wish in the future that the internet wouldn't remember or be able to be found.


----------



## Swift

mnsalem said:


> "completed ready for you to restore" ???
> 
> 
> So .. no news about the backups at alll? :/


Saw that and got really worried.

VPS up here but not restored or anything. Sent them a ticket, hopefully they answer with a specific answer rather than a generic copy and paste reply.


----------



## saltspork

My VPS on la-vps20 is up with my default root password, but imaged with CentOS 5 (which I had never used). It looks the same story for every other machine on the node, judging by the default Apache page. Better than nothing...

I haven't got any useful response to my fresh installation ticket yet, just two generic copy-pastes.


----------



## upsetcvps

ok so vps is back up but this is troubling: my ssh client is not warning me that the server's fingerprint has changed so I assume it matches what it was before the hack.  However, I also cannot log in using ssh keys so not everything is the same (and I can't seem to log in using passwords either...).  Can anyone provide some insight?


----------



## maounique

upsetcvps said:


> ok so vps is back up but this is troubling: my ssh client is not warning me that the server's fingerprint has changed so I assume it matches what it was before the hack.  However, I also cannot log in using ssh keys so not everything is the same (and I can't seem to log in using passwords either...).  Can anyone provide some insight?


Use solus console to login then change password and key.


----------



## upsetcvps

Mao said:


> Use solus console to login then change password and key.


Yes, once solusvm is up again I'll just be wiping the install completely, but I still don't understand the current behavior.


----------



## jfreak53

> As we finish up installing the new VPS' on the final machines I wanted to give an update. Within the next 5 hours, all OpenVZ VPS' should be installed and completed ready for you to restore.
> We still have a few Xen machines that had severe damage to them, we are still going to working on them and will be ready tomorrow. I will make a decision tonight about compesation and release another email in the morning. We would like to thank everyone once again for all your patience.
> 
> Here at ChicagoVPS we want to ensure this will never happen again in the future. We are in the progress of working closely with a security firm for a complete audit over our infrastructure.
> 
> Thank you
> 
> The ChicagoVPS Team


So basically they lied about everything, thanks Chris, and they have zero backups and we are on our own!  That's fine but they should have said that to begin with and I wouldn't have trusted their TOS saying they "had" backups. So now I've been lying to my customers this entire time saying the company had backups of some and they might get it ha ha ha Thanks a lot cVPS.


----------



## jfreak53

Not to mention I still have 3 VPS units still offline completely ha ha ha


----------



## jacobsta811

upsetcvps said:


> ok so vps is back up but this is troubling: my ssh client is not warning me that the server's fingerprint has changed so I assume it matches what it was before the hack.  However, I also cannot log in using ssh keys so not everything is the same (and I can't seem to log in using passwords either...).  Can anyone provide some insight?


The slices that are back up use whatever password *solusvm* has for you (nothing restored - they reinstalled over my chicago slice that was back down). So the password is whatever the hacked file says it is, ironically. IE if you changed your password using "passwd" they don't have it, so they couldn't reset it to that password, and they *didn't* reset all passwords even though they should; and I agree, I am not actually using these slices until I can reimage a fresh install on and change the password *immediately*. If you want to see what they brought up, dig up the initial email from when they provisioned you and it should have the password that is currently on the slice. Not sure why your key didn't change though - all of mine did; that might be a client issue on your end with it not alerting you.

My only VPS offline this morning is the one that I shut down, but all my Ubuntu nodes still fail apt-get so something is wrong with the nameservers in whatever image they were using.

Edit to add, no, they clearly had some backups from some time - my chicago node initially was restored intact it looked like, then taken down and later reimaged over with a fresh install. So they had some level of backups (for some nodes - not atlanta) but no procedure for restoring them in any mass way.


----------



## jfreak53

Well for me the passwords used at signup, I have them all stored, yet those are not it. So even though fresh installs are there I can't give passwords to my clients because I have no clue what they are ha ha. Great cVPS.


----------



## jer

VPS on Chi-VPS10 - still down, no ping.


----------



## bellicus

I'm on Chicago-44 and I've been down since then, And they can't seem to give me an honest answer about my vps or not..


----------



## upsetcvps

jacobsta811 said:


> Not sure why your key didn't change though - all of mine did; that might be a client issue on your end with it not alerting you.


I don't think it's a client issue.  The last write to my ~/.ssh/known_hosts was about a month ago, I've visually confirmed the fingerprint in ~/.ssh/known_hosts and the new one presented to me match, and my client alerts me if I change a key on a different server.


----------



## jer

I think cvps took today off.


----------



## Drar

jer said:


> I think cvps took today off.


^This...


----------



## bellicus

jer said:


> I think cvps took today off.


Would you put it past them? I'm still waiting for a simple reply :unsure: My 2nd CHI VPS just went dead and it didn't go down at all during the whole mess


----------



## upsetcvps

Yep, plane landed in Brazil by now.


----------



## Drar

You know what guys, I am starting to think that Chicago VPS have lost the backups or don't have any backups of our VPS at all.

I have several tickets opened asking about the about the data restoration but all I get are unrelated canned replies. I will not be surprised if on their next "RFO" they will say something like "Hey sorry but our backups got corrupted etc etc so we will just give you a fresh VPS so you can start from scratch yada yada yada..."

If they don't have backups they might as well come out and tell the truth so that we can have another contingency plan instead of waiting and relying on false hopes. I have couple GBs worth of data and it will be a pain in the a** to upload it using my home internet connection due to slow upload speeds.

Again, this is just me thinking about the "what ifs". Let's just hope that I am wrong about this...

What are your thoughts so far?

EDIT: Just got a confirmation from one of the users here that the data on his VPS has been restored by CVPS.

Will update this post as soon as my VPS has been restored as well.


----------



## zulualpha

I finally got one of my 2 VPS in buffalo back up today - it was a fresh install & the password hadn't been changed. There was no notification to let me know that it was back up. Fortunately I have my own backups, but it's going to be time consuming to upload them all.


----------



## Aldryic C'boas

Food for thought:  SolusVM has a backup system included.  However, information is stored much the same way as node information is - and if the attacker could wipe the nodes, there was nothing stopping him from wiping even remote backups as well if they were tied into Solus.

Sure, there's always the chance that they used different software, or wrote their own scripts for backups.  But I wouldn't wager on that.


----------



## jfreak53

I just got one of my 10 back up with the backup restored finally! But God it took forever, not to mention, it was still using the old password and they didn't even let me know it was back up!! Even though I have a TICKET open requesting this IP be restored ha ha. None others yet.


----------



## upsetcvps

I seem to be able to reset my root password through wmhcs now.  If your vps is up but you don't have log in credentials, I suggest you do this (and then change your password again once you are logged in using passwd).


----------



## jfreak53

Hahahahaha get this crap, just got it in a new message to a ticket I had open:



> At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.
> 
> We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.
> 
> You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).
> 
> Thank you again for your business and support.
> 
> ---------------
> Matthew
> Support Guru



Wait, what?! I thought in the last RFO report they said they had completed ALL node fixing and were now working on restoring? So which is a lie, this or that?


----------



## mnsalem

jfreak53 said:


> Hahahahaha get this crap, just got it in a new message to a ticket I had open:
> 
> Wait, what?! I thought in the last RFO report they said they had completed ALL node fixing and were now working on restoring? So which is a lie, this or that?


I got the same one, like, the EXACT wording .. so its another mass response, i was going to post it ... you beat me to it.

ya well, they're moving forward, a bit slow perhaps, but in the right direction ... i hope at least the backups are there. that's ALL i care about right now.


----------



## mnsalem

Just got a new email!



> With the recent SolusVM exploits that have affected our company and others with a negative impact, many of our customers and us are not supportive of enabling public facing access to our SolusVM VPS CP as additional code could be exploitable. Let's not take a risk when it comes to security. At this time, we are releasing an alternative frontend solution to our customers to allow them to reboot, start, shut down, serial console, change root pass, or change hostname on their VPS. We hope to be making this more feature rich soon, however at the moment the only thing that you CANNOT do with this new frontend is: reinstall VPS, manage DNS entries, or create central backup. We are working on making these features available to you ASAP.
> 
> You can now access your virtual server controls at https://billing.chicagovps.net/clientarea.php?action=products . Select the service, and under the "Virtual Server Control" section you can manage multiple aspects of your VPS, including reboot, start, shut down, serial console, change root password, or change hostname.
> 
> No client's VPS data was leaked or accessed by a 3rd party during this hack. The hacker(s) did not directly access any VPS container or hypervisor, and simply used a SolusVM exploit to wipe out and cause damage to a certain number of VPS nodes. The intentions of the malicious hackers was cause mayhem within our company by wiping some of our servers. With this compromise, our SolusVM database was accessed by a third party. As such, there is a possibility that any passwords that were related with SolusVM could be at risk, for example your initial password you signed up with. For those clients VPS's that are now accessible and showing as an online state in the virtual server controls section in our client area, we urge that you immediately change your root password by clicking on the "Change Root Password" button.
> 
> Let it be clear that this compromise did not impact our client area in anyway, so any billing information, etc stored in our client area at billing.chicagovps.net is safe.
> 
> For good measure, please take a minute to change your client area password. Those who used the same SolusVM password as the client area should do this promptly. https://billing.chicagovps.net/clientarea.php?action=changepw
> 
> On a related note, rest assured we're making great progress in our recovery. A further update regarding this matter will be sent out later today.
> 
> We thank our customers for their continued support during this ordeal.
> 
> Regards,
> 
> ChicagoVPS Team


----------



## drvelocity

So it took them 5 f*ing days to just get a bunch of blank nodes up and running from scratch?  I could have done that singlehandedly in a day... ridiculous.


----------



## Aldryic C'boas

> No client's VPS data was leaked or accessed by a 3rd party during this hack.


Umm...what?  The Solus master has direct access to every node - so yes, if the hacker knew of someone already at CVPS they wanted data of, it would be as simple as sending a command like *tar zxvf /var/www/data.tgz /vz/private/<vservers.ctid>/* to the node, then grabbing the tarball of that VPS's data from the node's webserver.  To be quite honest, you should simply assume that someone has all of your VPS' contents from prior to the hack, and take the appropriate security precautions.


----------



## sleddog

Someone help me, I don't understand:

Your VPS goes down

You enquire to the company

Based on the response, or lack of, you make a decision

Wait...

Restore elsewhere from backups

CVPS is a budget company. Don't expect the world on a budget.


----------



## mnsalem

> once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take *3-4 days* before all requests have been fulfilled.


3 to 4 days????? what the fluff???



Remind me again, how did *Magnificent Nick* from RamNode do that in less than 2 days?????


----------



## drvelocity

CVPS 3-4 days = Probably not going to happen.  They just need more time to think about how to slowly equivocate until their customers just go away quietly.  Step 2: Restart the entire company under a new name!


----------



## MannDude

I'm sort of curious of the specs of their backup nodes. They've got 4 backup servers listed in Solus to handle backing up 109~ nodes, minus the Atlanta location. By the sounds of it, there is more than just Atlanta that wasn't backed up. LA, Chicago and Buffalo appear to be the only physical locations with backup servers, though they could of course be transferring data from Dallas to LA or to Chicago or something aswell I suppose.


----------



## upsetcvps

Aldryic C said:


> Umm...what?  The Solus master has direct access to every node - so yes, if the hacker knew of someone already at CVPS they wanted data of, it would be as simple as sending a command like *tar zxvf /var/www/data.tgz /vz/private/<vservers.ctid>/* to the node, then grabbing the tarball of that VPS's data from the node's webserver.  To be quite honest, you should simply assume that someone has all of your VPS' contents from prior to the hack, and take the appropriate security precautions.


Not to mention solusvm has the ability to change passwords right?  So if one controls solusvm, ...



drvelocity said:


> CVPS 3-4 days = Probably not going to happen.  They just need more time to think about how to slowly equivocate until their customers just go away quietly.  Step 2: Restart the entire company under a new name!


Yep.  Most users won't end up asking for a restore.  Those that do will just happen to be the ones that had corrupted or wiped backups.  How unlucky for them!


----------



## Aldryic C'boas

upsetcvps said:


> Not to mention solusvm has the ability to change passwords right?  So if one controls solusvm, ...


True enough.  What really should concern people is the database leak, though.  *@* could confirm this... but wasn't the DB leak dated at least a day or more before the actual attack?  The guy could've grabbed any number of VPS dumps in the meantime before tearing everything up.


----------



## jer

Matthew || Staff  Saturday, June 22nd, 2013 (15:08)

At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.

We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.

You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).

Thank you again for your business and support.
---------------
Matthew
Support Guru

Jer S || Client  Saturday, June 22nd, 2013 (09:42)

Hey folks, still down. No ping.

I do not need a restore. I've looked at my rows in the leaked database, my container parameters don't seem broken.
----------------------------

Chris's response makes it sound like I want a restore. I don't. I don't know how to type it clearer.

So if you go in the Client Area to Services > Manage My VPS   - - it's broken.

Servers still down.


----------



## vkimball

I had 3 of my 5 vps' affected by the CVPS incident.

2 of them (in Atlanta) were back up on Thursday with a clean install of Ubuntu 12.04.1 and no nameservers defined.  Easy enough to change my root password, configure nameservers and update to current.

The other one (in LA) was back up on Friday with a clean CentOS 5.8 install rather than Ubuntu 12.04.  Unfortunately, I can't reinstall my vps because WHMCS doesn't allow that function.  Guess I'll have to open a ticket and wait.

I don't really care about backups because I was only using them as test environments and all the important data was stored elsewhere.


----------



## MannDude

jer said:


> Matthew || Staff  Saturday, June 22nd, 2013 (15:08)
> 
> At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.
> 
> 
> We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.
> 
> 
> You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).
> 
> 
> Thank you again for your business and support.
> 
> 
> ---------------
> 
> 
> Matthew
> 
> 
> Support Guru
> 
> Jer S || Client  Saturday, June 22nd, 2013 (09:42)
> 
> Hey folks, still down. No ping.
> 
> 
> I do not need a restore. I've looked at my rows in the leaked database, my container parameters don't seem broken.
> 
> 
> ----------------------------
> 
> Chris's response makes it sound like I want a restore. I don't. I don't know how to type it clearer.
> 
> So if you go in the Client Area to Services > Manage My VPS   - - it's broken.
> 
> Servers still down.


Matthew, I am 98% sure is indeed Chris. Just an alias.


----------



## upsetcvps

http://www.facebook.com/l.php?u=https%3A%2F%2Fbilling.chicagovps.net%2Fclientarea.php%3Faction%3Dchangepw&h=[omitted]&s=1

Can anyone explain to me why the "change your password" link in their latest e-mail goes through facebook...?


----------



## jer

Chris is forcing me to have to go do this.. http://www.youtube.com/watch?v=LWFZwjtHqKU (that's me and my kissy - might be fun to watch if you aren't doing anything), instead of working on coding. Damn it.


----------



## mikho

Aldryic C said:


> Food for thought: SolusVM has a backup system included. However, information is stored much the same way as node information is - and if the attacker could wipe the nodes, there was nothing stopping him from wiping even remote backups as well if they were tied into Solus.
> 
> 
> Sure, there's always the chance that they used different software, or wrote their own scripts for backups. But I wouldn't wager on that.


I guess this is what happened to Virpus back in 2011 when 19 of their nodes where taken out as in complete wipe with no backups.


----------



## srichter

I'm not sure why they keep responding to the tickets with canned responses (well I assume they hope we give up). They say that it's possible for them to restore from backups, but then when you request that you just get a canned response. Why not say "We have added you to the queue" and mark the ticket "On Hold" or "In Progress" instead of just sending out another canned email? How many times am I going to have to request they restore from the backup?



> At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.
> 
> We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.
> 
> You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).
> 
> Thank you again for your business and support.


----------



## drvelocity

srichter said:


> I'm not sure why they keep responding to the tickets with canned responses (well I assume they hope we give up). They say that it's possible for them to restore from backups, but then when you request that you just get a canned response. Why not say "We have added you to the queue" and mark the ticket "On Hold" or "In Progress" instead of just sending out another canned email? How many times am I going to have to request they restore from the backup?


Amen - the whole thing is so hilariously ridiculous.  The worst possible result one could ever expect in a situation like this.  Anyone who actually gets a backup restored please make sure to let us know here, because my best guess is that it's all total BS.


----------



## Lanarchy

upsetcvps said:


> http://www.facebook.com/l.php?u=https%3A%2F%2Fbilling.chicagovps.net%2Fclientarea.php%3Faction%3Dchangepw&h=[omitted]&s=1
> 
> Can anyone explain to me why the "change your password" link in their latest e-mail goes through facebook...?


He posted to FB first, then just copy/paste to the email, including the FB outgoing link.

My stats, I have not opened a ticket asking for any restores.

NY - good, original data ... an hour ago randomly messed up the firewall but I did csf -r and all is well again. Probably coincidence, but who knows at this point.

CHI - good, original data

LA - good, original data

ATL - one fresh install and accessible, one unknown and serial console says 'console configuration not found' and will not boot.

However, for safety, once we get a real control panel, I will reinstall all of them fresh. Just in case any passwords were saved, or any files inserted (which noone has mentioned yet, but I assume is a possibility)


----------



## srichter

I replied to their canned response with



> I understand you may have other things to work on before restoring from the backup, but this is my request that you do so. Please do not mark this ticket as answered, please insert me into the queue for my data to be restored as soon as it is possible for you to do so.
> 
> "File restoration is possible, though must be done manually by our staff." - Please do so
> 
> "We are happy to restore your files" - Please do so
> 
> Thank You


And they replied



> Steve,
> 
> You got it.
> 
> ---------------
> Matthew
> Support Guru


And changed the status of the ticket to "Restoration."

That's a good sign!


----------



## mnsalem

Same here! Ticket just got marked for "Restoration" in red!


----------



## vkimball

vkimball said:


> The other one (in LA) was back up on Friday with a clean CentOS 5.8 install rather than Ubuntu 12.04.  Unfortunately, I can't reinstall my vps because WHMCS doesn't allow that function.  Guess I'll have to open a ticket and wait.


Well, I'm happy to report that my LA vps was just reinstalled as Ubuntu 12.04.


----------



## upsetcvps

Can anyone that has a fresh debian (squeeze) install on a cvps openvz container, post their ssh server public key (or fingerprint)?  The fingerprint is what you see when you connect to a server for the first time to verify its identity.  You can obtain the fingerprint by running the command: 

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key


----------



## Tactical

I like to throw this out there, all the griping about canned responses. How would you respond if you had 2000 ppl saying the same thing? The exact same way. I wish everyone the best of luck.  The lesson is to back up your own data. Peace out


----------



## upsetcvps

SgtZinn said:


> I like to throw this out there, all the griping about canned responses. How would you respond if you had 2000 ppl saying the same thing? The exact same way. I wish everyone the best of luck.  The lesson is to back up your own data. Peace out


They should display the canned response on the page where people submit tickets.  That way if the canned response actual answers your question you don't even create a ticket!

But I imagine the griping is because the canned response doesn't actually address the issue in the ticket.  Which means you have to open the ticket again until someone bothers to actually read what you typed.


----------



## Tactical

*I* see your point. Its a big mess but there main priority is get the nodes up then try to work off the tickets. So that is probably y ppl are getting canned responses. It just takes time. It could take weeks to get it all worked out, but I'm no expert.


----------



## Hugohp

I just got this email from them. 



> Hello,
> 
> Unfortunately backups for your container from our master backup repository are not available. If you utilized our free Central Backup feature to create a restore point for your service we can backup from that data. If you did not utilize that free service we do not have backups and will be unable to restore any of your data.
> 
> 
> ---------
> Luc Ayotte
> ChicagoVPS Support Tech
> [email protected]


No backup for me......


----------



## leeboof

Hugohp said:


> I just got this email from them.
> 
> No backup for me......


When did you request your restore? I wonder how far down the list I am.


----------



## Hugohp

leeboof said:


> When did you request your restore? I wonder how far down the list I am.


I requested it on June 19th, but they put me on "Restoration" status today (June 22nd in México) and after 8 hours I received that email.


----------



## helobye

Anyone else have this problem? CVPS have configured an empty Ubuntu 12.04 container for me on LA-18, but the VM is unable to access the internet, and I can't SSH in. I added a few DNS servers and changed the hostname (using the Serial Console) to no avail. Iptables is empty.


----------



## srichter

Hugohp said:


> I just got this email from them.
> 
> No backup for me......


What location/node were you on?


----------



## drvelocity

My account was just restored, and the data looks like it's from June 10 - (I'm on a NY server).  I have to say I thought this was going to drag on for more than a week - they just made my weekend.  I have to eat my previous words, they came through for me on this one, I very sincerely hope that everyone else affected gets all fixed up as well.  Good luck all!  And don't forget to back up!


----------



## helobye

Just got this notice of no backups, node LA-18: 



> Hello,
> 
> Unfortunately backups for your container from our master backup repository are not available. If you utilized our free Central Backup feature to create a restore point for your service we can backup from that data. If you did not utilize that free service we do not have backups and will be unable to restore any of your data.
> 
> 
> ---------
> Luc Ayotte
> ChicagoVPS Support Tech
> [email protected]


----------



## MannDude

It's somewhat interesting seeing the contrast between the updates in this thread and the updates in the LET thread.


----------



## Aldryic C'boas

MannDude said:


> It's somewhat interesting seeing the contrast between the updates in this thread and the updates in the LET thread.


tl;dr for those of us that don't go to LET? :3


----------



## Hugohp

srichter said:


> What location/node were you on?


I don't remember the exact node and doesn't appear on WHCMS, but location is Chicago.


----------



## leeboof

Just got the reply no backups for me either. I'm guessing the list of nodes that were not backed up was correct that was posted earlier. Might help to check that for an idea if you are still waiting.

EDIT: Here is the post: http://vpsboard.com/topic/758-chicagovps-cvps-hacked-new-solusvm-exploit/?p=12538


----------



## srichter

> Hello,
> 
> 
> Your VPS has been re-installed from our backup's.


I'm on buf-17. Looks to be from 6/14.

Good luck to everoyne else.


----------



## whatever

they emailed me today



> Hello,
> 
> Unfortunately backups for your container from our master backup repository are not available. If you utilized our free Central Backup feature to create a restore point for your service we can backup from that data. If you did not utilize that free service we do not have backups and will be unable to restore any of your data.
> 
> 
> ---------
> Luc Ayotte
> ChicagoVPS Support Tech
> [email protected]


my node in LA,I think they don't care for Mexico, terrible situation (MX customer as Hugohp)


----------



## helobye

leeboof said:


> Just got the reply no backups for me either. I'm guessing the list of nodes that were not backed up was correct that was posted earlier. Might help to check that for an idea if you are still waiting.
> 
> EDIT: Here is the post: http://vpsboard.com/topic/758-chicagovps-cvps-hacked-new-solusvm-exploit/?p=12538


I'm not so sure, node LA-18 is not on the list and I was told the node has no backups.


----------



## helobye

whatever said:


> they emailed me today
> 
> my node in LA,I think they don't care for Mexico, terrible situation (MX customer as Hugohp)


Sorry to hear your node was unable to be restored as well. If your containers IP address is in the 198.46.137.x range it's node LA-18 like mine.


----------



## whatever

helobye said:


> Sorry to hear your node was unable to be restored as well. If your containers IP address is in the 198.46.137.x range it's node LA-18 like mine.


Yes, I am in the same range, I had a hope from them when I saw "We are happy to restore your files..", but I 'll have to do everything again. Long nights since today,


----------



## drmike

Aldryic C said:


> What really should concern people is the database leak, though.  @buffalooed could confirm this... but wasn't the DB leak dated at least a day or more before the actual attack?  The guy could've grabbed any number of VPS dumps in the meantime before tearing everything up.


The database for SolusVM was borrowed on the 17th.

RamNode was hit by this Solus issue a day or more before this.

RamNode got taken down Sunday morning 9AM or earlier Eastern time on Sunday, June 16, 2013.

ChicagoVPS noticed their hack and node damage around 2AM Eastern time on Monday, June 17th, 2013.  

That means the exploit existed and was known for some 17 hours between these events.


----------



## drmike

MannDude said:


> They've got 4 backup servers listed in Solus to handle backing up 109~ nodes, minus the Atlanta location. By the sounds of it, there is more than just Atlanta that wasn't backed up. LA, Chicago and Buffalo appear to be the only physical locations with backup servers


 

Yeah well, they aren't backing everything up.

Backing all this up across the internet = mega slow process in mass.

I'm pretty sure if you are on one of the following nodes, your data is gone for good:

 

+--------+-----------+-----------------------------+

| nodeid | name      | hostname                    | number of vservers on nod |

|      1 | localhost | manage.chicagovps.net       |

|     35 | chi22     | chi-vps22.chicagovps.net    | 10 vservers

|     21 | chi10     | chi-vps10.chicagovps.net    | 23 vservers

|     25 | chi13     | chi-vps13.chicagovps.net    | 11 vservers

|     31 | chi18     | chi-vps18.chicagovps.net    | 17 vservers

|     37 | chi24     | chi-vps24.chicagovps.net    | 11 vservers

|     39 | chi23     | chi-vps23.chicagovps.net    | 15 vservers

|     42 | chi27     | chi-vps27.chicagovps.net    | 16 vservers

|     79 | chissd1   | chi-ssd-vps1.chicagovps.net | 59 vservers

|     48 | chi32     | chi-vps32.chicagovps.net    |  13 vservers

|     49 | chi33     | chi-vps33.chicagovps.net    |  0 vservers

|     57 | chi40     | chi-vps40.chicagovps.net    | 23 vservers

|     65 | chi47     | chi-vps47.chicagovps.net    | 26 vservers

|     68 | chi50     | chi-vps50.chicagovps.net    | 29 vservers

|     76 | chi51     | chi-vps51.chicagovps.net    | 21 vservers

|     80 | chi53     | chi-vps53.chicagovps.net    | 80 vservers

|    109 | atl1      | atl-vps1.chicagovps.net     | 161 vservers

|    110 | atl2      | atl-vps2.chicagovps.net     | 183 vservers

|    128 | atl3      | atl-vps3.chicagovps.net     |  56 vservers

|    131 | atl4      | atl-vps4.chicagovps.net     | 122 vservers

|    133 | atl5      | atl-vps5.chicagovps.net     | 92 vservers

|    138 | atl6      | atl-vps6.chicagovps.net     | 109 vservers

|    148 | nj1       | nj-vps1.chicagovps.net      | 13 vservers

|    149 | dfw1      | dfw-vps1.chicagovps.net     | 5 vservers

|    150 | njkvm1    | nj-kvm-vps1                 | 3 vservers

|    151 | chi70     | chi-vps70.chicagovps.net    | 31 vservers

+--------+-----------+-----------------------------+

 

Simply said, those 26 servers were not configured for FTP backups via Solus.

 

There are roughly 104 nodes that were live.  26 un-backedup servers represents an even 25% of their servers that weren't being backed up.


----------



## srichter

buffalooed said:


> ChicagoVPS noticed their hack and node damage around 2AM Eastern time on Monday, June 17th, 2013.


My VPS didn't go down until ~3AM EST on the 18th


----------



## Swift

> At this time virtually all customers are back online; some with all original files in tact and others with new containers. File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.
> We are happy to restore your files, though if its easier and quicker for you to restore the files yourself from your own backups we recommend going in that direction.
> 
> You now have full control over your VPS from WHMCS (you can stop/start/restart/reset root PW/reload it, etc).
> 
> Thank you again for your business and support.
> 
> ---------------
> 
> Matthew
> 
> Support Guru


 I hate myself for not having a recent backup.


----------



## jer

Josh Aborad || Staff  Saturday, June 22nd, 2013 (20:34)

Jer,
We will investigate this issue for you and report back when we have more information for you. Thank you for your patience.
Josh
Support Guru

 

 

Yada yada... just don't know what to say.


----------



## jer

(double post)


----------



## drmike

srichter said:


> My VPS didn't go down until ~3AM EST on the 18th



That's very strange.  Know I saw others say Tuesday they went down.  Pretty sure that was the admins trying to fix something and probably declaring the node unsalvageable, necessitating a full server rebuild.


----------



## srichter

buffalooed said:


> That's very strange.  Know I saw others say Tuesday they went down.  Pretty sure that was the admins trying to fix something and probably declaring the node unsalvageable, necessitating a full server rebuild.


The 18th was Tuesday


----------



## drmike

srichter said:


> The 18th was Tuesday


Yes, indeed.  Odd to have nodes go down 24 hours+ after the attack.  Wondering if things were left running on some of these servers from the attack.

I'd be mighty concerned if I were on a node that did this on Tuesday.


----------



## mnsalem

Finally!y VPS is back online! 3 hours after my ticket was out into restoration category! The backup I think is around June 10th to June 14th .. On BUF19


But hey, at least they did have backups after all! 


Good luck to everyone else getting your data back!


----------



## mnsalem

the latest email received from CVPS - Report #10



> First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario. We are absolutely dedicated to ensuring the happiness and recovery of every single impacted customer.
> 
> Now that we’ve begun to get ahead of the support load I’m able to share more specific details on our current status. At this time all customer services should be online and available. A small subset of customers may still be offline as their containers will require further investigation and attention. A few nodes have been condensed/transferred/migrated so please do not be alerted if your node ID has changed. Direct access to SolusVM remains inactive as we wait for their internal and external security audits to be completed (as discussed here: http://www.lowendtalk.com/discussion/11327/solusvm-audit-update#latest). During the interim you are able to control your containers through our billing/support system and may request OS reloads via ticket.
> 
> Initially the mitigation strategy called for all impacted nodes to be inspected for logs, reloaded, and then images from our backup database installed. This process proved far too timely and dramatically slowed down the recovery effort which is also partially to blame for our poor communication during the first 48 hours post-compromise. We altered the plan two days ago, deciding to instead install all impacted customer service from scratch; this revised strategy has allowed us to rapidly return to full online status and reduce pressure as we work diligently to recover data for all customers who require it. ChicagoVPS has two separate backup facilities, a free public facing system called Central Backup and a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted. For customers on those nodes, if you ran a Central Backup your data is absolutely safe and you may request a restoration via ticket. Customers who were not on the nodes with corrupted secondary backups can request a restoration regardless of whether they utilized the free Central Backup feature. The restoration process currently has a 6-12 hour lead time once you’ve requested it.
> 
> ChicagoVPS will be analyzing this event closely so we may implement refined plans to both protect against repeat issues and to ensure our communication and reaction strategies are improved. While SolusVM has released multiple updates in the past few days to fix vulnerabilities which allowed this event, and others, to occur we take full responsibility for our outage. We simply should have reacted more quickly, more effectively and provided better communication. We’ve already learned from this past week, and we will continue to learn.
> 
> As we push towards final resolution on all fronts we will continue to update our customers via email, Facebook (https://www.facebook.com/chicagovps) and the “Low End Talk” message board (http://www.lowendtalk.com/discussion/11304/chicagovps-update#latest).
> 
> Thank you again for your patience, loyalty and understand. We appreciate you.
> 
> Regards,
> 
> Chris Fabozzi
> Director of Operations
> ChicagoVPS


----------



## drmike

Fabozzi never fails to amaze me with his abuse of the English language.



mnsalem said:


> First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario.


 

He calls this *SITUATION *a _scenario_? Nothing imagined about it.  Nothing planned.  It is pure post-apocalypse mopping of the floors.  Scenario would be appropriate if he had contingency planning and policies in place which foresaw and had resolution for such an outcome or scenario to happen in the future potentially.








 







> ​
> 
> 
> 
> 
> 
> 
> 
> sce·nar·io
> _noun_ \sə-ˈner-ē-ˌō, _US also and especially British_ -ˈnär-\
> 
> _plural_ *sce·nar·i·os*
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Definition of SCENARIO
> 
> 1
> 
> a *:* an outline or synopsis of a play; _especially_ *:* a plot outline used by actors of the commedia dell'arte
> 
> 
> b *:* the libretto of an opera
> 
> 
> 2
> 
> a *:* screenplay
> 
> 
> b *:* shooting script
> 
> 
> 3
> 
> *:* a sequence of events especially when imagined; _especially_ *:*an account or synopsis of a possible course of action or events <his _scenario_ for a settlement envisages…reunification — Selig Harrison>
> 
> 
> 
> 
> 
> See scenario defined for English-language learners »
> 
> 
> 
> See scenario defined for kids »
> 
> 
> 
> 
> Examples of SCENARIO
> 
> 
> A possible _scenario_ would be that we move to the city.
> The most likely _scenario_ is that he goes back to school in the fall.
> The best-case scenario would be for us to finish the work by tomorrow.
> In the worst-case scenario, we would have to start the project all over again.
> 
> 
> 
> 
> Origin of SCENARIO
> Italian, from Latin _scaenarium_ place for erecting stages, from _scaena_ stage
> First Known Use: 1875
> 
> 
> 
> 
> 
> Related to SCENARIO
> 
> 
> Synonyms
> 
> screenplay, script
> 
> 
> 
> 
> 
> Related Words
> 
> shooting script; story, text








 












mnsalem said:


> a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted.


Small group of nodes?  It's roughly 25% that weren't running backups plus whatever others that had failed/wrong/broke backups.   Small is north of 25%?!?!?!  Pretty lousy numbers from the son of an accountant.

I HATE HATE HATE providers that smokescreen things and put fluff all over the place.  Chris wasn't busy doing the admin work.  He should have been replying to tickets and doing the press / update / news circuit (Facebook, Twitter, LET, etc.)

Maybe the third hack-go-round CVPS will handle it properly  ?


----------



## MannDude

Ok, so this is the original thread and IPB staff has restored it. Due to some errors going to have to split the threads up I think. There may be duplicate content in this thread that also exists in this thread here: http://vpsboard.com/topic/758-chicagovps-cvps-hacked-new-solusvm-exploit/

That thread will be locked until I clean it up.


----------

