What could be the cause for auditd to use almost 400% CPU?

[Kokaku Kidotai]

Good day gentlemen,

What could be the reason for the "auditd" process to use almost 400% (roughly 397%) for over 425 hours (431 hours before being suspended)? The process "auditd" was called by the system user "Apache". The box is running CentOS 6 with zPanel, two sites and a private PPTP VPN.

top:



Any idea where to start the investigation from? I totally have no idea but I want to solve this issue.

Yours faithfully,

Kokaku Kidotai

 

[serverian]

It's a malicious process. Kill it and secure your /tmp

 

[Kokaku Kidotai]

http://linux.die.net/man/8/auditd

The Linux Audit Daemon is a malicious process? You may be right, but I really don't know. I find it strange enough that Apache has actually started it and not root. Does zPanel actually require audit? I am having the feeling that hackers may have compromised the zPanel installation.

How can I find out and prove that it really was hacked or is a malicious process? Where to look for logs?

 

[serverian]

ps aux | grep auditd and you'll see it's running from /tmp

Yes, your server is compromised somehow.

You shouldn't run zpanel in the first place.

 

[Kokaku Kidotai]

It isn't my server. I am just the only administrator on duty so I have to do everything and this includes taking care of abuse, too. The provider reported it to us and we suspended the VPS and now I want to find out why and what.

I'm going to unsuspend it and look into it.

 

[HostSailor]

It could be a DDoS tool, a port scanning tool or any other malicious tool disguised under a wrong name I guess.

 

[Patrick]

zPanel is the issue, it's vulnerable and you just got hacked because of it.


Backup and reinstall with another panel

 

[Kokaku Kidotai]

As the VPS was suspended (shutdown, made unusable and unaccessable) the /tmp directory is totally empty and after unsuspension the process is not running with the parameters as in the screenshot above.

There are many SSH break in attempts in the secure files in /var/log.

Well, thanks guys. My thoughts were right I guess. I had this feeling all the time that zPanel was compromised and the server is being abused through that.

 

[blergh]

As the VPS was suspended (shutdown, made unusable and unaccessable) the /tmp directory is totally empty

Yes, this is because most distro's clear it upon reboot/boot and suspending/shutting it down will clear it.

 

[DomainBop]

It's a malicious process. Kill it and secure your /tmp

Killing it and securing the temp will take care of the high load and prevent the attacker from running auditd but the VPS user is probably still fucked even if they remove ZPanel because if the exploit allowed the attacker to run auditd it probably also allowed the attacker to access the auditd logs. The auditd logs contain a lot of information that could be used by an attacker to gain complete control of a server, install backdoors, etc..

After killing the process, securing /tmp, and removing zPanel do a thorough security check of the system and/or do a fresh reinstall (and do a security check of any data that is restored from a backup for backdoors, rootkits, etc that may exist) , change passwords, etc. 

The Linux Audit Daemon is a malicious process?

Auditd is a valuable tool for system administrators but in the wrong hands the log info can be used for malicious purposes.

One final thought... all the containers on an openvz node share the same kernel and auditd's operations are tightly integration with the kernel (kernel logging, system calls, bla bla bla are all recorded) so...

 

[Deleted]

strace the process, or attach gdb to it and see what it's doing

 

[Magiobiwan]

It's not the REAL auditd, it's a malicious process using that name running out of /tmp. zPanel is your culprit. I'm not sure what it does, but it's not what auditd does. Those processes USED to be named "ksoftirqx", but their names changed recently.