VPS's .. The dangers of doing business with idiots

[Deleted]

I was doing some consulting work for a small VPS provider (who isn't on this forum, as far as I know), they wanted a custom kernel module written to log performance metrics. This was to be installed on the host node, so it could audit things like to see how 'busy' a vm was from the host node (and perhaps track abuse ;-)

When I initially logged into the server, the users had a webserver, mail, rsync's running to an NFS partition on a different block of IPs in another datacenter.

While I was attempting to write this module, I needed some things to test it out on, anyways, the client of mine decided to tell me that his webserver was acting fishy with his VM frontend (whatever run-of-the-mill control panel) by redirecting to certain websites with no apparent source.


Alas, after doing to debugging with GDB/IDA, his server was compromised by a malicious httpd binary designed to remain hidden; Not only that, his sshd was compromised as well (it wasn't logging connections anymore). This is probably a new version of Cdorked floating around (didn't investigate much)

Now, I informed the client of this, and his response was: "Oh [we] removed the ssh hacks off months ago, but it keeps coming back" (not exactly, but that sums it up)

I had most of the module written, but at this point, I refunded him his initial payment and told him 'no thanks' because of the attitude.

This is a reminder to all of you that sell VPS's, if you're not careful or paranoid about security, you'll end up compromised, and your clients can have their data stolen as well which will come back to bite you 100,000 times harder.

 

[Magiobiwan]

Hell, if your box has been compromised in ANY WAY, it's a Nuke and Pave. That's the ONLY way you'll be sure to get whatever rootkits or other malware got in out.

 

[Deleted]

Exactly.

These people that run these so-called VPS companies scare me to death by not being more paranoid about security. I've had things hacked before, due to 0 day exploits (wayyy back in the 90s, qpopper was a big target), solution was to reformat and reinstall.

But we have the Microsoft generation, combined with being lazy and not wanting to do it. 

 

[HaitiBrother]

If your a VPS provider and your box gets compromised you don't deserve to be a vps provider....

 

[MCH-Phil]

If your a VPS provider and your box gets compromised you don't deserve to be a vps provider....

If you think your security / polices / etc are impenetrable....

 

[DomainBop]

This is a reminder to all of you that sell VPS's, if you're not careful or paranoid about security, you'll end up compromised, and your clients can have their data stolen as well which will come back to bite you 100,000 times harder.

If they're a low end clown outfit like ChicagoVPS, VPS Ace, et al, then they have nothing to worry about if they're hacked multiple times because LowEndBox will still post their offers even after multiple hacks, and will still post their offers even when the company fails to notify their customers of a database breach or fails to follow the legally required notification procedures for data breaches, and Maarten the low end admin will not mention to consumers that the provider he is featuring has been recently hacked and had their entire customer database posted on the Internet because the host is "a victim" and shouldn't be penalized for something that was "completely out of their control"

For me, the difference between a pump-and-dump and a host that got hacked is that a host that got hacked doesn’t choose for it. It happens to them; they don’t see it coming. It’s something mostly or completely out of their control (depending on their security). How the backlash of such an event is handled is another thing. I don’t know what happened at VPS Ace nor have I ever seen any notice about it. IRC is not the most trustworthy source. That’s no attack on you, but I hope you know what I mean: people are weird at IRC sometimes.

We’ve seen many hosts hacked over the past couple of years and I think it could be grounds for caution, but not for ignoring them. I’ve decided not to mention hacks in offers for a while. Why? The damage has already been done, it has been dealt with and any host would surely prevent such a thing from happening again (to the best of their ability).

Let me ask you this: would you keep mentioning it in offers? If yes, why? What’s the benefit for the customer?

http://lowendbox.com/blog/vps-ace-12year-128mb-6month-2gb-openvz-vps-and-more-in-the-usa-and-the-netherlands/

 

[HaitiBrother]

If you think your security / polices / etc are impenetrable....

If you think security is real....

 

[Royal]

So who do guy's think is the safest vps provider?  :mellow:

 

[peterw]

So who do guy's think is the safest vps provider?  :mellow:

I don't know any safe vps provider. Nobody has 100% uptime and nobody does is save because humans do make errors. The supplier of electric energy, the datacenter, staff or attackers hacking and ddosing a provider.

 

[fixidixi]

I think the only question is how a company treat people if such thing happens (got hacked etc).

-communication

-changing~reconsidering security policies

-regular audits

etc.

 

[Deleted]

How about some common sense for starters. 

I would not trust anyone who doesn't have a dedicated 'security team' [sic] or at least someone with a clue. Having low level tech support reinstall the OS on hacked boxes is pretty pathetic, since the intrusion point is not known. Tracking down HOW is more important than anything, at least to me.

 

[kaniini]

Well, this is why I say using SSH for transport in control panels (in the way these control panels set it up, anyway) is a bad idea.  In many cases, the rootkit probably gets in through the SSH credentials in the panel, typically leaked through some sort of SQLi vulnerability.

Instead, mechanism should be entirely separated from transport.  If you are using SSH as transport, the SSH user should be restricted to a special shell locking it to the endpoint.  This is considered a best practice since forever, yet all of the panels I see using SSH aren't doing it.

And really, SSH isn't necessarily the best transport for this sort of thing... a distributed bus with a required proof-of-work signature scheme has a lot of useful properties that SSH does not, mostly that it's lightweight and ultimately transport-agnostic.  Hmm, this sounds familiar.

Instead of worrying about vulnerabilities and exploit response, simply reduce the attack surface to something that you can formally prove is correct.  Then use proven security measures like grsecurity to secure the control plane.

 

[Aldryic C'boas]

Common sense, which should be the first line of defence and security, is what I often notice a dearth of.  Way back when, I posted a guide for extended WHMCS login tracking, two people actually PM'd me (unprompted mind, and out of the blue) with the root login credentials of the boxes they were running WHMCS on;  one asking that I setup said tracking for him, another that actually sent me his root SQL pass as well, wanting custom work done.

 

[drmike]

I don't see why the client should have been discontinued and refunded really... 

Seems like the client could benefit from more knowledge and consulting by you @.

Security is a big laundry list of things....  

Real companies, running real businesses, employ people to deal just with this aspect of the business --- just like they employ finance folks, marketing people, janitors, etc.

There is WAY too much emphasis on people being know-it-alls in these small companies.  That's just darn near impossible.

Certainly would like to see more consultants with businesses handling things like you were Monk and helping with security practices and better configs.

 

[HaitiBrother]

Common sense, which should be the first line of defence and security, is what I often notice a dearth of.  Way back when, I posted a guide for extended WHMCS login tracking, two people actually PM'd me (unprompted mind, and out of the blue) with the root login credentials of the boxes they were running WHMCS on;  one asking that I setup said tracking for him, another that actually sent me his root SQL pass as well, wanting custom work done.

If these hosts exist still, please list their names if you remember them...

 

[Aldryic C'boas]

One is still around - but no, I don't break confidentiality.  I spent some time impressing the values of security and caution; hopefully that was enough for them to learn from the mistake.
 

I don't see why the client should have been discontinued and refunded really...

Honestly, I would've done the same.  I decline doing work for someone if I feel there's a security or compromise risk - that's just asking to be blamed for any problems that surface down the road.  Speaking strictly in doing approved contract work for other companies, that is - we take the time to educate our clients and help them resolve security issues.

 

[KuJoe]

I don't see why the client should have been discontinued and refunded really...

I would have done the same also. If he did do the work and they got hacked again (which seems inevitable) then they can blame him for it and that would be both a PR nightmare for his reputation and a potential legal one.

It would be like suing a roofing company because they were the last ones to touch your roof before it collapsed when you didn't bother getting a load bearing wall repaired after somebody drove a car into it. While that might seem far-fetched, it's still costly for the roofing company to defend themselves and there's a really good chance they business will be impacted by it being associated with a collapsed roof.

 

[marlencrabapple]

It gets worse when you realize that this kind of attitude isn't just limited to idiot VPS providers, but to most of the world. I got a job doing Wordpress development for some guys that did lawyer marketing, and just about EVERY client they had was running a two year old version of wordpress on an unsecured virtual server (often times root had an FTP account with a 5 character password) with an old version of PHP, Apache, etc. I ended up reading some old tickets for our own VPS somewhere down the line, and apparently even we had been compromised multiple times, to the point where we'd been suspended and had to big for forgiveness from the provider. Blows my mind that I, a glorified intern, cared more about basic security (don't use root, choose long/random passwords, update your software, etc.) than they did.

 

[DomainBop]

Common sense, which should be the first line of defence and security, is what I often notice a dearth of.

Many VPS providers suffer from a lack of common sense but just as importantly due to push button solutions like SolusVM, CPanel/WHM, etc many VPS providers, and small hosting providers in general, have limited technical knowledge of virtualization, Linux, etc (sorry but I don't consider 2-3 years of using Linux as a second OS or running a Minecraft server to be adequate experience) and almost zero knowledge of how to secure a server so even if they had common sense their lack of experience would still make their hosting service and nodes walking time bombs.  Case in point from WHT (prediction: offer on LET within 3 months from this guy )

i want to sell vps. but dont know how to create a vps. if i install kvm on my dedicated server it will be done or i have to install any other thing?

 

[tchen]

So, who's got remote logging turned on and tripwires in place?