GreenValueHost forced password reset - Security breach?

[drmike]

Well, like all web stuff, logging is on...  That would be within WHMCS as well as underlying web server logs.

I try to always be straight with info and honest with folks as to what they are seeing. No real time to vet this info to death like normally.  So things to note, live time:

1. The screencaps above were from NOON not midnight.

2. NOON correlates to when IRC posting happened - not claiming it has any relationship big picture. Suspect just A+B + bad timing.

3. Original emails started shooting out at midnight, by others accounts overnight.

4. There were more emails after NOON event.

 

[hellogoodbye]

What did the logs look like from the original midnight event? Were the same IPs behind the emails, or did it show different IPs?

 

[drmike]

Another screencap, with activity on one of the IPs - this is from the web server logs:

 

[jarland]

Another screencap, with activity on one of the IPs - this is from the web server logs:

Odd no POST in there, unless it's elsewhere.

 

[drmike]

Note the runscope part...

https://www.runscope.com/

"Testing on your terms.

Create tests based on actual calls made from within your apps. Capture traffic from any app without changing your code. Define test criteria with flexible assertions and variables to pass state between requests."

API debugging outsourced service.

Definitely sounds / seems like hacking at URL construction.

 

[drmike]

And note @jarland, they were 200 OK then went 404... That was when rules were put into place and firewall erected (i.e. directory blocked externally).

 

[HH-Josh]

Issue resolved as regards to my account. Appears that someone in the office signed up using my details for a service we never used, hence why my email is listed with GVH - as stated before I've never personally been a client so seemed a bit strange as to why I was listed, more to do with a staff member in the office not communicating with me.


GVH now seem to think I have a vendetta against them to try and ruin their reputation, which is completely false and un true. I never have, or will be a client of GVH, so it would be completely unfair to judge them in any way.


Hope to hear more of an update shortly though as regards to what's been going on - quite an interesting situation.


- Josh

 

[raindog308]

I'm thinking that perhaps I want to add a CSF rule for WHMCS...too many password reset requests from an IP in X time is a temp block.

 

[Nett]

No it was not a security breach. Client data is completely safe and has not been leaked. We will have a final statement sent out regarding this issue soon.

How could you tell?

 

[DomainBop]

And note @jarland, they were 200 OK then went 404... That was when rules were put into place and firewall erected (i.e. directory blocked externally).

It's a little late to be putting firewall rules in today when they've been in operation for 2 years. They should have secured that directory when they installed WHMCS (not to mention changing the admin folder name).  The WHMCS documentation also suggests moving the crons folder to a non publically accessible folder, password protecting the admin directory, limiting access by IP, etc.  http://docs.whmcs.com/Further_Security_Steps#Change_your_WHMCS_Admin_Folder_Name

The screenshot basically confirm what we all know: The kids at GVH aren't the brightest bulbs on the security block, and failed to properly secure their site and as a result they put their users info at risk .  I would ask if they do regular PCI/vulnerability scans of their WHMCS and SolusVM installations but I think I already know the answer to that one: NO.

 

[jarland]

It's a little late to be putting firewall rules in today when they've been in operation for 2 years. They should have secured that directory when they installed WHMCS (not to mention changing the admin folder name). The WHMCS documentation also suggests moving the crons folder to a non publically accessible folder, password protecting the admin directory, limiting access by IP, etc. http://docs.whmcs.com/Further_Security_Steps#Change_your_WHMCS_Admin_Folder_Name


The screenshot basically confirm what we all know: The kids at GVH aren't the brightest bulbs on the security block, and failed to properly secure their site and as a result they put their users info at risk . I would ask if they do regular PCI/vulnerability scans of their WHMCS and SolusVM installations but I think I already know the answer to that one: NO.

Meh, you know PCI compliance and most vulnerability scans are useless and serve mostly to milk money from those who don't know any better. In this market it's both easier and more beneficial to just not be stupid and read documentation...which so many people just can't seem to be bothered to do.

 

[drmike]

Best practices are swell.  But shit happens for stupidity sake and for plain ole never read the farking manual.

I have no defense of the actions or lack thereof.  Seen bad outcomes from compliant and non compliant folks.

Even if wide open and cron.php right out front, should require credentials and other stuff.  There is some disconnect or gap between what happened and what we all think about how WHMCS works.   Wish I had the answer on this one.

A ticket exists at WHMCS for the matter.

I believe in March release, WHMCS had update(s) to the cron.php file.

GVH is running licensed version and current.

So... ahh yeah... if this is real issue, everyone wide open could be smacked in same way.  Folks ought to get to securing installs and generally other communities should be made aware of this issue.

 

[SkylarM]

So... ahh yeah... if this is real issue, everyone wide open could be smacked in same way.

While there is SOME fault to WHMCS for it being so easy to run crons/pop importer email files, etc the real blame lies in anyone dumb enough to not secure their admin directory. WHMCS even suggests some basic securing of the admin directory: http://docs.whmcs.com/Further_Security_Steps

 

[jarland]

It's really incredible what you can do with a little apache and some .htaccess. If you're willing, you can effectively mod_rewrite yourself out of most common uses for mod_sec, and I wish more people understood apache doesn't execute a php cron in your crontab, it doesn't need to be accessible via apache, or whatever other hipster thing you've got listening on 80.

 

[DomainBop]

Meh, you know PCI compliance and most vulnerability scans are useless and serve mostly to milk money from those who don't know any better

They're not as effective as practicing basic security on a daily basis but quarterly scans are a requirement for (Internet) merchant accounts [yes, I realize most low end providers don't have merchant accounts].  Sometimes they might even discover a vulnerability that the site owner missed. As far as cost, the daily scans are a money cow for the scan providers but every merchant account provider I've used has thrown in the quarterly scans for free so it is possible to get scanned semi-regularly without spending anything . Comodo's hackerguardian also has a "5 free scans over 3 IPs" free trial so anyone could have their site scanned for vulnerabilities before opening to the public.

just not be stupid and read documentation...which so many people just can't seem to be bothered to do.

looks at GVH and I'm sure many more "push button" script users (in all online industries)..

the real blame lies in anyone dumb enough to not secure their admin directory.

The basic "further security steps" suggested by WHMCS would take at most 5 minutes and are so basic that even the most technically inept person could do them so there is really no excuse for not doing them.

 

[jarland]

Well I gotta say props to GVH for sharing screenshots of whmcs log and access logs. That's so much more than I'd expect of some others. Maybe this issue is more complex than it looks. No excuses for some security failures but if I asked everyone who has made a mistake or oversight to raise their hand, I would have mine up as well.

 

[GVH-Jon]

We have identified the root cause and will be releasing a public explanation statement shortly.

 

[drmike]

This is precursor to their coming out party for this issue.

There was no hack.   Nothing was compromised.  Nothing more than cron.php being public and when accessed set off a job in there.

Call it bad configuration + a queued marketing email campaign with a bad toggle option.

Fire of request to cron.php and presto, emails go out in mass... Every time.

 

[Kris]

Fire of request to cron.php and presto, emails go out in mass... Every time.

Are you sure? Have you been able to duplicate this, or speculation? 

The logs you posted was from mid-day, when GET (not post) was hitting cron.php. Maybe an angry ex-employee with API details.

In terms of the scanner hitting them 12 hours after the emails, seems like a nice way to build evidence for their story that it was cron... by scanning your own site. 

Things have seemed a bit off since you 'helped and were chatting with HVH' and the CC goons, now you're the go-to for GVH help, defending. 

I have to ask... How much was your price? 

EDIT: From the explanation below, seems feasible actually. Good on GVH for a fast investigation. 

 

[GreenHostBox]

Got this in my email below. I guess this is solved now.

As you are aware, GreenValueHost experienced issues in the past 24 hours where multiple password resets were sent to customers. We apologize for the flurry of emails.

GreenValuehost was NOT hacked, there wasn't a security compromise either.  Customers are safe and secure.

What happened involves the WHMCS billing panel.

Two files: cron.php and /admin were accessible to the public.  These should have been secured with additional rules.  Yes, we have since added multiple layers of security to protect these files from public access.

A marketing email was pre-wrote and placed in WHMCS for a 2014 Spring Overstock Sale.  The default template was erroneously set to / left at “Automated Password Reset”.  WHMCS defaults the default template pull down option to “Automated Password Reset”.

Marketing emails run from cron.php events.   Therefore, when someone accessed the cron.php file it triggered the sending of the marketing email, which was then set as “Automated Password Reset”.  This happened a total of 17 times, generating 17 emails per email account.

GreenValueHost debugged this issue by manually running the secured cron.php, analyzing a few emails sent and looking in WHMCS.

We have a ticket with WHMCS which will be appended to reflect the debugging and resolution with recommendations to prevent this in future with other WHMCS users (beyond simply securing said files).

We welcome any customers concerned about the matter or who may be experiencing password problems to submit a ticket.


Thank You
GreenValueHost Team