amuck-landowner

Random process name sending DDoS

mrwright

mrwright

New Member
Maybe someone here could shed some light on a problem I have been seeing.

Of course, the system is compromised, and the only true solution here is to wipe and reload. However, I have been seeing more and more of it recently and would like to understand how this is happening.

First off; no unknown users or ssh logins. And this is the true problem for myself.

This has not happened to any of my production systems; most sit behind firewalls anyways. But in the case of publicly facing VPS slices and dedicated systems I been seeing more and more.

Ramon process, no idea where is coming from, called fgrmlhgxlf


top - 13:33:18 up 2 min, 1 user, load average: 0.52, 0.26, 0.10
Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32837716k total, 394212k used, 32443504k free, 9948k buffers
Swap: 4188668k total, 0k used, 4188668k free, 60320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1751 root 20 0 77288 5248 200 S 1.7 0.0 1:40.59 fgrmlhgxlf
1 root 20 0 19232 1488 1220 S 0.3 0.0 0:00.80 init

So the first thing, of course, kill it. BUT a new process of the same kind appears now called  bfqirtcejh


[root@test ~]# kill -9 1751
[root@test ~]# top
top - 13:34:35 up 3 min, 1 user, load average: 0.18, 0.21, 0.09
Tasks: 230 total, 1 running, 229 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.1%us, 2.3%sy, 0.0%ni, 90.5%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Mem: 32837716k total, 439044k used, 32398672k free, 9952k buffers
Swap: 4188668k total, 0k used, 4188668k free, 67688k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2601 root 20 0 197m 13m 196 S 61.3 0.0 0:02.00 bfqirtcejh
1 root 20 0 19232 1496 1220 S 0.0 0.0 0:00.81 init

Digging more into the process itself (note: same type of process but different event named cixtwrhpet)


[root@test tmp]# lsof | grep cixtwrhpe
cixtwrhpe 1715 root cwd DIR 9,2 4096 2 /
cixtwrhpe 1715 root rtd DIR 9,2 4096 2 /
cixtwrhpe 1715 root txt REG 9,2 625718 4328099 /usr/bin/cixtwrhpet
cixtwrhpe 1715 root 0u CHR 1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 1u CHR 1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 2u CHR 1,3 0t0 3829 /dev/null
cixtwrhpe 1715 root 3u IPv4 13426 0t0 TCP 172-245-111-138-host.colocrossing.com:46763->103.240.141.54:6002 (ESTABLISHED)
cixtwrhpe 1715 root 4u raw 0t0 14453 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 5u raw 0t0 14454 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 6u raw 0t0 14455 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 7u raw 0t0 14456 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 8u raw 0t0 14457 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 9u raw 0t0 14458 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 10u raw 0t0 14459 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 11u raw 0t0 14460 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 12u raw 0t0 14461 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 13u raw 0t0 14462 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 14u raw 0t0 14463 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 15u raw 0t0 14464 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 16u raw 0t0 14476 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 17u raw 0t0 14477 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 18u raw 0t0 14478 00000000:00FF->00000000:0000 st=07
cixtwrhpe 1715 root 19u raw 0t0 14479 00000000:00FF->00000000:0000 st=07

Using the source above I was interested in what kind of executable this might be:


[root@test tmp]# file /usr/bin/cixtwrhpet
/usr/bin/cixtwrhpet: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped

How about network process, this was from the bfqirtcejh process, note the same pid " 2601/cat resolv.con"


[root@test ~]# netstat -antop
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1654/sshd off (0.00/0/0)
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1732/master off (0.00/0/0)
tcp 0 1 172.245.111.138:45799 103.240.141.54:6002 SYN_SENT 2627/netstat -antop on (0.64/0/0)
tcp 0 64 172.245.111.138:22 98.159.218.194:62690 ESTABLISHED 2143/sshd probe (0.45/0/0)
tcp 0 300 172.245.111.138:45800 103.240.141.54:6002 ESTABLISHED 2613/netstat -an on (0.54/0/0)
tcp 0 0 172.245.111.138:45313 103.240.141.54:6002 ESTABLISHED 2601/cat resolv.con keepalive (43.08/0/0)
tcp 1 0 172.245.111.138:45798 103.240.141.54:6002 CLOSE_WAIT 2617/bash keepalive (59.62/0/0)
tcp 0 1 172.245.111.138:45797 103.240.141.54:6002 SYN_SENT 2609/whoami on (0.33/1/0)
tcp 0 1 172.245.111.138:45794 103.240.141.54:6002 SYN_SENT 2622/id on (0.10/1/0)
tcp 0 300 172.245.111.138:45791 103.240.141.54:6002 ESTABLISHED 2612/pwd probe (1.33/0/0)
tcp 0 0 :::22 :::* LISTEN 1654/sshd off (0.00/0/0)
tcp 0 0 ::1:25 :::* LISTEN 1732/master off (0.00/0/0)

Even more weird 103.240.141.54 is owned by ClearDDoS Technologies in Hong Kong.

Anyone have an idea how this trojan like pos got installed without access and/or any more knowledge on it. My searching has come up with almost nothing. Maybe some kind of Fork bomb but still doesn't explain how it gets introduced into the system initially.

Thanks!
 
MartinD

MartinD

Retired Staff
Verified Provider
Retired Staff
Running kloxo by any chance?

I've seen a lot of that. Some common patterns;

1) Check for cronjobs that respawn it

2) Check binaries for common bash utils in /usr/bin and /usr/sbin - I've seen them being changed.

3) A lot are run from /tmp and then deleted. Disable execution on /tmp
 
Last edited by a moderator:
mrwright

mrwright

New Member
wlanboy said:
This won't help if you do not limit the amount of failed logins per minute / per ip.
Click to expand...
Fair enough. Anything in production has been hardened. I have only seen this on new fresh installs that have not been locked down but left sitting overnight.

MartinD said:
Running kloxo by any chance?
Click to expand...
Nope, fresh CentOS minimal.

Thanks for the suggestions. If/when I see this again ill take a look. I did however see one executable running in tmp.
 
MartinD

MartinD

Retired Staff
Verified Provider
Retired Staff
One last thought - is your install source (template/iso) trustworthy?
 
mrwright

mrwright

New Member
MartinD said:
One last thought - is your install source (template/iso) trustworthy?
Click to expand...
Indeed. CentOS ISO, net install using the centos.org mirror.

I haven't seen anything this destructive in a hijacked iso before but I guess I couldn't put it past someone.
 
MartinD

MartinD

Retired Staff
Verified Provider
Retired Staff
Unfortunately I have though I can't remember the source.

If you think about it, it makes perfect sense. 'Infecting' the host at the point of installation is the best way to create a huge botnet with very little effort.
 
J

joepie91

New Member
I've seen this on a system that got owned through an SSH bruteforce (on a non-22 port). It tries to wipe its own logs, so that could be why you're not seeing any evidence of intrusion. Fortunately, the particular sample I encountered wasn't very successful at it, and thus left a trace.

Also, please disable password login on your VPS. Only ever use keypair authentication, system-wide.
 
Last edited by a moderator:
P

perennate

New Member
Verified Provider
SolusVM said:
Here you go: https://blog.avast.com/2015/01/06/linux-ddos-trojan-hiding-itself-with-an-embedded-rootkit/
Click to expand...
Doesn't seem related, whatever infected OP's machine didn't try to hide itself at all.

If the password was randomly generated (assuming it was long enough) then it's very unlikely that was the attack vector, even offline attacks to get sixteen random characters are not feasible. But maybe you have a sudo account on the system with weak password? It'd be difficult to gain root access via most services.
 
Last edited by a moderator:
Francisco

Francisco

Company Lube
Verified Provider
perennate said:
Doesn't seem related, whatever infected OP's machine didn't try to hide itself at all.

If the password was randomly generated (assuming it was long enough) then it's very unlikely that was the attack vector, even offline attacks to get sixteen random characters are not feasible. But maybe you have a sudo account on the system with weak password? It'd be difficult to gain root access via most services.
Click to expand...
Because OP's VPS is OpenVZ. It's XOR.DDOS.

The way the system works is pretty crazy:

- Infect server

- Call home reporting kernel version

- If C&C has the sources to that kernel, send down a pre-compiled kernel module

--- If it doesn't, try to pull the sources from some where (be it yum/apt-get, or even from /usr/src)

- Download a unique compiled binary for that infected server to get past basic hash signatures

- pound Tsunmai SYN floods right away

While Aldryic wrote a pretty sweet system to handle brute forces over all of our locations, there's still people that use really bad passwords or they get exploited through websites.

At this point I've written some iptables rules to handle the outbound flood so it doesn't cause issues for other people on the node.

Francisco
 
Last edited by a moderator:
mrwright

mrwright

New Member
Francisco

This event specifically was just a dedicated system.

However on my OpenVZ nodes Nodewatch suspends the VPS within seconds. The affected customer is then asked to reload and to choose a more aggressive password.
 
T

TheLinuxBug

New Member
In my experience if this was a Debian openVZ template you used to setup the server, especially an older  Debian 7 or Debian 6 image there were some of the OpenVZ templates that had a really old version of openssh default installed which was vulnerable to exploit.  I had a few occasions where similar things happened to me from hosts using outdated templates, I would order the server it would be provisioned while I was asleep and by the time I woke up the server would be suspended.  I would login on console to review the server before re-installing to find somehow they had gained access via ssh using the challangeresponse auth method.  It seems this in older versions of OpenSSH was exploitable to gain access.  I have also seen some other variants but bottom line is the provider needs to update their templates and makes sure the packages in the templates are up to date so as not to be provisioning exploitable servers.

There are of course other types of attacks, but the OPs situation sounds similar to a few experiences I have had with smaller hosts who don't seem to keep things updated.

my 2 cents.

Cheers!
 
Munzy

Munzy

Active Member
It sounds like you are having a bot hit your ssh and or exploiting an issue.

After getting any server, the first thing you should do is an apt-get update && apt-get upgrade -y or the centos yum update and yum upgrade. Then a restart.
 
wlanboy

wlanboy

Content Contributer
Francisco said:
Because OP's VPS is OpenVZ. It's XOR.DDOS.


While Aldryic wrote a pretty sweet system to handle brute forces over all of our locations, there's still people that use really bad passwords or they get exploited through websites.
Click to expand...
As @Munzy says. There are a lot of outdated images flying aroung.

And autostarting vps won't help either.

5 out of 10 vps I ordered through the last two years where outdated and did have a running ssh and apache webserver with really bad default settings. 

Ordered them, went to work, coming home to see a vulnerable server.
 
You must log in or register to reply here.
Top
amuck-landowner