# Random process name sending DDoS



## mrwright (Jul 13, 2015)

Maybe someone here could shed some light on a problem I have been seeing.

Of course, the system is compromised, and the only true solution here is to wipe and reload. However, I have been seeing more and more of it recently and would like to understand how this is happening.

First off; no unknown users or ssh logins. And this is the true problem for myself.

This has not happened to any of my production systems; most sit behind firewalls anyways. But in the case of publicly facing VPS slices and dedicated systems I been seeing more and more.

Ramon process, no idea where is coming from, called *fgrmlhgxlf*


top - 13:33:18 up 2 min, 1 user, load average: 0.52, 0.26, 0.10
Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32837716k total, 394212k used, 32443504k free, 9948k buffers
Swap: 4188668k total, 0k used, 4188668k free, 60320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1751 root 20 0 77288 5248 200 S 1.7 0.0 1:40.59 fgrmlhgxlf
1 root 20 0 19232 1488 1220 S 0.3 0.0 0:00.80 init

So the first thing, of course, kill it. BUT a new process of the same kind appears now called  *bfqirtcejh*


[[email protected] ~]# kill -9 1751
[[email protected] ~]# top
top - 13:34:35 up 3 min, 1 user, load average: 0.18, 0.21, 0.09
Tasks: 230 total, 1 running, 229 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.1%us, 2.3%sy, 0.0%ni, 90.5%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Mem: 32837716k total, 439044k used, 32398672k free, 9952k buffers
Swap: 4188668k total, 0k used, 4188668k free, 67688k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2601 root 20 0 197m 13m 196 S 61.3 0.0 0:02.00 bfqirtcejh
1 root 20 0 19232 1496 1220 S 0.0 0.0 0:00.81 init

Digging more into the process itself (note: same type of process but different event named *cixtwrhpet)*


[[email protected] tmp]# lsof | grep cixtwrhpe
cixtwrhpe 1715 root cwd DIR 9,2 4096 2 /
cixtwrhpe 1715 root rtd DIR 9,2 4096 2 /
cixtwrhpe 1715 root txt REG 9,2 625718 4328099 /usr/bin/cixtwrhpet
cixtwrhpe 1715 root 0u CHR  1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 1u CHR 1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 2u CHR 1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 3u IPv4 13426 0t0 TCP 172-245-111-138-host.colocrossing.com:46763->103.240.141.54:6002 (ESTABLISHED)
cixtwrhpe 1715 root 4u raw 0t0 14453 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 5u raw 0t0 14454 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 6u raw 0t0 14455 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 7u raw 0t0 14456 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 8u raw 0t0 14457 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 9u raw 0t0 14458 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 10u raw 0t0 14459 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 11u raw 0t0 14460 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 12u raw 0t0 14461 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 13u raw 0t0 14462 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 14u raw 0t0 14463 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 15u raw 0t0 14464 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 16u raw 0t0 14476 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 17u raw 0t0 14477 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 18u raw 0t0 14478 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 19u raw 0t0 14479 00000000:00FF->00000000:0000 st=07

Using the source above I was interested in what kind of executable this might be:


[[email protected] tmp]# file /usr/bin/cixtwrhpet
/usr/bin/cixtwrhpet: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

How about network process, this was from the *bfqirtcejh *process, note the same pid " 2601/cat resolv.con"


[[email protected] ~]# netstat -antop
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1654/sshd off (0.00/0/0)
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1732/master off (0.00/0/0)
tcp 0 1 172.245.111.138:45799 103.240.141.54:6002 SYN_SENT 2627/netstat -antop on (0.64/0/0)
tcp 0 64 172.245.111.138:22 98.159.218.194:62690 ESTABLISHED 2143/sshd probe (0.45/0/0)
tcp 0 300 172.245.111.138:45800 103.240.141.54:6002 ESTABLISHED 2613/netstat -an on (0.54/0/0)
tcp 0 0 172.245.111.138:45313 103.240.141.54:6002 ESTABLISHED 2601/cat resolv.con keepalive (43.08/0/0)
tcp 1 0 172.245.111.138:45798 103.240.141.54:6002 CLOSE_WAIT 2617/bash keepalive (59.62/0/0)
tcp 0 1 172.245.111.138:45797 103.240.141.54:6002 SYN_SENT 2609/whoami on (0.33/1/0)
tcp 0 1 172.245.111.138:45794 103.240.141.54:6002 SYN_SENT 2622/id on (0.10/1/0)
tcp 0 300 172.245.111.138:45791 103.240.141.54:6002 ESTABLISHED 2612/pwd probe (1.33/0/0)
tcp 0 0 :::22 :::* LISTEN 1654/sshd off (0.00/0/0)
tcp 0 0 ::1:25 :::* LISTEN 1732/master off (0.00/0/0)

Even more weird 103.240.141.54 is owned by ClearDDoS Technologies in Hong Kong.

Anyone have an idea how this trojan like pos got installed without access and/or any more knowledge on it. My searching has come up with almost nothing. Maybe some kind of Fork bomb but still doesn't explain how it gets introduced into the system initially.

Thanks!


----------



## Nett (Jul 13, 2015)

mrwright said:


> First off; no unknown users or ssh logins. And this is the true problem for myself.


Is the root password something easy to guess? Attackers do very well to remove their footprint.


----------



## mrwright (Jul 13, 2015)

@Nett

Nope. Random from keepass.


----------



## drmike (Jul 13, 2015)

103.240.141.54

That is a Command Control IP for a botnet.

It appears and is wrote in detail about over here:

http://blog.level3.com/security/breaking-botnets-how-level-3-and-cisco-worked-together-to-improve-the-internets-security-and-stop-sshpsychos/

Dated: April 9, 2015

May be related to this:

http://blog.malwaremustdie.org/2014/09/mmd-0028-2014-fuzzy-reversing-new-china.html


----------



## wlanboy (Jul 14, 2015)

mrwright said:


> @Nett
> 
> Nope. Random from keepass.


This won't help if you do not limit the amount of failed logins per minute / per ip.


----------



## MartinD (Jul 14, 2015)

Running kloxo by any chance?

I've seen a lot of that. Some common patterns;

1) Check for cronjobs that respawn it

2) Check binaries for common bash utils in /usr/bin and /usr/sbin - I've seen them being changed.

3) A lot are run from /tmp and then deleted. Disable execution on /tmp


----------



## mrwright (Jul 14, 2015)

wlanboy said:


> This won't help if you do not limit the amount of failed logins per minute / per ip.


Fair enough. Anything in production has been hardened. I have only seen this on new fresh installs that have not been locked down but left sitting overnight.



MartinD said:


> Running kloxo by any chance?


Nope, fresh CentOS minimal.

Thanks for the suggestions. If/when I see this again ill take a look. I did however see one executable running in tmp.


----------



## MartinD (Jul 14, 2015)

One last thought - is your install source (template/iso) trustworthy?


----------



## mrwright (Jul 14, 2015)

MartinD said:


> One last thought - is your install source (template/iso) trustworthy?


Indeed. CentOS ISO, net install using the centos.org mirror.

I haven't seen anything this destructive in a hijacked iso before but I guess I couldn't put it past someone.


----------



## MartinD (Jul 14, 2015)

Unfortunately I have though I can't remember the source.

If you think about it, it makes perfect sense. 'Infecting' the host at the point of installation is the best way to create a huge botnet with very little effort.


----------



## joepie91 (Jul 14, 2015)

I've seen this on a system that got owned through an SSH bruteforce (on a non-22 port). It tries to wipe its own logs, so that could be why you're not seeing any evidence of intrusion. Fortunately, the particular sample I encountered wasn't very successful at it, and thus left a trace.

Also, *please* disable password login on your VPS. Only ever use keypair authentication, system-wide.


----------



## mrwright (Jul 14, 2015)

joepie91 said:


> Also, *please* disable password login on your VPS. Only ever use keypair authentication, system-wide.


Everything in production uses ssh keys.

This was a simple install without anything hardened yet.


----------



## SolusVM (Jul 14, 2015)

Here you go: https://blog.avast.com/2015/01/06/linux-ddos-trojan-hiding-itself-with-an-embedded-rootkit/


----------



## SolusVM (Jul 14, 2015)

Check the yum log for things like zmap being installed and take a peek in /etc/init.d and /boot


----------



## perennate (Jul 15, 2015)

SolusVM said:


> Here you go: https://blog.avast.com/2015/01/06/linux-ddos-trojan-hiding-itself-with-an-embedded-rootkit/


Doesn't seem related, whatever infected OP's machine didn't try to hide itself at all.

If the password was randomly generated (assuming it was long enough) then it's very unlikely that was the attack vector, even offline attacks to get sixteen random characters are not feasible. But maybe you have a sudo account on the system with weak password? It'd be difficult to gain root access via most services.


----------



## Francisco (Jul 15, 2015)

perennate said:


> Doesn't seem related, whatever infected OP's machine didn't try to hide itself at all.
> 
> If the password was randomly generated (assuming it was long enough) then it's very unlikely that was the attack vector, even offline attacks to get sixteen random characters are not feasible. But maybe you have a sudo account on the system with weak password? It'd be difficult to gain root access via most services.


Because OP's VPS is OpenVZ. It's XOR.DDOS.

The way the system works is pretty crazy:

- Infect server

- Call home reporting kernel version

- If C&C has the sources to that kernel, send down a pre-compiled kernel module

--- If it doesn't, try to pull the sources from some where (be it yum/apt-get, or even from /usr/src)

- Download a unique compiled binary for that infected server to get past basic hash signatures

- pound Tsunmai SYN floods right away

While Aldryic wrote a pretty sweet system to handle brute forces over all of our locations, there's still people that use really bad passwords or they get exploited through websites.

At this point I've written some iptables rules to handle the outbound flood so it doesn't cause issues for other people on the node.

Francisco


----------



## mrwright (Jul 15, 2015)

Francisco

This event specifically was just a dedicated system.

However on my OpenVZ nodes Nodewatch suspends the VPS within seconds. The affected customer is then asked to reload and to choose a more aggressive password.


----------



## TheLinuxBug (Jul 15, 2015)

In my experience if this was a Debian openVZ template you used to setup the server, especially an older  Debian 7 or Debian 6 image there were some of the OpenVZ templates that had a really old version of openssh default installed which was vulnerable to exploit.  I had a few occasions where similar things happened to me from hosts using outdated templates, I would order the server it would be provisioned while I was asleep and by the time I woke up the server would be suspended.  I would login on console to review the server before re-installing to find somehow they had gained access via ssh using the challangeresponse auth method.  It seems this in older versions of OpenSSH was exploitable to gain access.  I have also seen some other variants but bottom line is the provider needs to update their templates and makes sure the packages in the templates are up to date so as not to be provisioning exploitable servers.

There are of course other types of attacks, but the OPs situation sounds similar to a few experiences I have had with smaller hosts who don't seem to keep things updated.

my 2 cents.

Cheers!


----------



## Munzy (Jul 15, 2015)

It sounds like you are having a bot hit your ssh and or exploiting an issue.

After getting any server, the first thing you should do is an apt-get update && apt-get upgrade -y or the centos yum update and yum upgrade. Then a restart.


----------



## wlanboy (Jul 16, 2015)

Francisco said:


> Because OP's VPS is OpenVZ. It's XOR.DDOS.
> 
> 
> While Aldryic wrote a pretty sweet system to handle brute forces over all of our locations, there's still people that use really bad passwords or they get exploited through websites.


As @Munzy says. There are a lot of outdated images flying aroung.

And autostarting vps won't help either.

5 out of 10 vps I ordered through the last two years where outdated and did have a running ssh and apache webserver with really bad default settings. 

Ordered them, went to work, coming home to see a vulnerable server.


----------



## Flapadar (Jul 16, 2015)

XOR.DDoS is a nasty one. We've got bruteforce protection in place _and_ filter outbound attacks.


The "Tsunami" attack in particular managed to avoid our existing filters the first time we saw it. Easily enough blocked, but by that point there had been a 900mbps attack for about half an hour. Horrid stuff.


----------



## ChrisB (Jul 16, 2015)

Francisco said:


> Because OP's VPS is OpenVZ. It's XOR.DDOS.
> 
> 
> The way the system works is pretty crazy:
> ...


Yeah I've been seeing a lot of these and we confirmed today that just by going through a few of them that they'll hammer servers non stop until they bruteforce the password and get in - as soon as it gets in, all incorrect root login attempts basically stop. 

Something worth noting is it seems to favour IP's it has successfully bruteforced in the past. It's a pretty intelligent system.


----------



## Munzy (Jul 17, 2015)

wlanboy said:


> As @Munzy says. There are a lot of outdated images flying aroung.
> 
> And autostarting vps won't help either.
> 
> ...


The defaults actually pissed me off so bad that I built a whole script to remove the shit, patch it, secure it, and a few other basic tasks.

https://cdn.content-network.net/sc/deb-quick-install.sh


```
https://cdn.content-network.net/sc/deb-quick-install.sh
```


----------

