Hetzner has been compromised

[WelltodoInformalCattle]

I received the following e-mail from Hetzner and was also linked to it by someone in #vpsboard:

Dear Client

At the end of last week, Hetzner technicians discovered a "backdoor" in one
of our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration
interface for dedicated root servers (Robot) had also been affected. Current
findings would suggest that fragments of our client database had been copied
externally.

As a result, we currently have to consider the client data stored in our Robot
as compromised.

To our knowledge, the malicious program that we have discovered is as yet
unknown and has never appeared before.

The malicious code used in the "backdoor" exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
been affected.

The standard techniques used for analysis such as the examination of checksum
or tools such as "rkhunter" are therefore not able to track down the malicious
code.

We have commissioned an external security company with a detailed analysis of
the incident to support our in-house administrators. At this stage, analysis
of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database
as Hash (SHA256) with salt. As a precaution, we recommend that you change your
client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card type
and the expiry date are saved in our systems. All other card data is saved
solely by our payment service provider and referenced via a pseudo card number.
Therefore, as far as we are aware, credit card data has not been compromised.

Hetzner technicians are permanently working on localising and preventing possible
security vulnerabilities as well as ensuring that our systems and infrastructure
are kept as safe as possible. Data security is a very high priority for us. To
expedite clarification further, we have reported this incident to the data
security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in
regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and
trust in us.

A special FAQs page has been set up at
http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further
enquiries.

Kind regards

Martin Hetzner

Hetzner Online AG
Stuttgarter Str. 1
91710 Gunzenhausen / Germany
Tel: +49 (9831) 61006-1
Fax: +49 (9831) 61006-2
[email protected]
http://www.hetzner.com


Register Court: Registergericht Ansbach, HRB 3204
Management Board: Dipl. Ing. (FH) Martin Hetzner
Chairwoman of the Supervisory Board: Diana Rothhan

 

[drmike]

Hetzner's write up is what I'd like to see other providers describing when compromised (the nature, some details, cleanup and post-event actions).

Compare this to what we've seen from all the other providers like CVPS.

As for Hetzner, doubt they have one compromise and this is likely the start of multiple successful attacks.

 

[jarland]

Very detailed and professional. Hope they manage to get a good picture of how it originally infected the system and share the information with the developers who can patch it up quickly.

 

[WelltodoInformalCattle]

Compare this to what we've seen from all the other providers like CVPS.

I'm all for CVPS-bashing but you really can't compare CVPS with Hetzner to be honest.

 

[drmike]

I'm all for CVPS-bashing but you really can't compare CVPS with Hetzner to be honest.

 No doubt, two different entities all together.

Just pointing to the right way to deal with issues vs. the patently wrong way.  Textbook style examples of the bookends.

 

[wlanboy]

Perfect reaction. I really want to know how that could happen. This is really a first class hack.

Still waiting for the smart guys to find a way to detect binaries that are only modified in RAM.