amuck-landowner

Security incident at OVH

vld

vld

New Member
Verified Provider
Hello,


A few days ago, we discovered that the internal security of our offices in Roubaix had been compromised. After internal investigations we found that a hacker was able to gain access to an email account of one of our system administrators. With this email access, they was able to gain access to the internal VPN of another employee. Then with this VPN access, they was able to compromise the access of one of the system admins who deals with the internal backoffice.


Internal security was based on 2 levels of verification:


- The requirement to be in the office or use the VPN, i.e.: the IP source


- The personal password


After this hack, we changed the internal security rules immediately:


- Passwords of all employees were regenerated for all types of access.


- We set up a new VPN in a secure PCI-DSS room with highly restricted access


- Consulting internal emails is now possible from the office / VPN


- All those who have critical access now have 3 verification levels:


- Ip source


- Password


- The staffs hardware token (YubiKey)


During the internal investigation into the security incident, we have discovered that hackers have probably gained privileged access for two actions:


- Recover the database of our customers in Europe


- Gain access to the installation server system in Quebec


The European customer database includes personal customer information such as: surname, first name, nic, address, city, country, telephone, fax and encrypted password.


The encryption password is based on SHA512, i.e.: it is very strong. It takes a lot of technical know-how to find the word password clearly. But it is possible. This is why we advise you to change the password for your user name. An email will be sent today to all our customers explaining the security incident and inviting them to change their password. Information on credit cards have not been consulted or copied. Today we do not store this on our infrastructure.


On the Quebec server system installation, the risk we have identified is that if the client had not withdrawn our SSH server key, the hacker could connect from your system and retrieve the password stored in the .p file. The SSH key is not usable from another server, only our Quebec backoffice . Therefore, where the client has not removed our SSH key and has not changed their root password, we immediately changed the password of the servers in the BHS DC to cancel this risk there. An email will be sent with the new password today. The SSH key will now be systematically erased at the server delivery end in Quebec as well as in Europe. If the client needs OVH for support, a new SSH key will need to be reinstalled.


Overall, in the coming months the back office will be under PCI-DSS which will allow us to ensure that the incident related to a specific hack on specific ndividuals will have no impact on our databases. In short, we were not paranoid enough so now we're switching to a higher level of paranoia. The aim is to guarantee and protect your data in the case of industrial espionage that would target people working at OVH.


We also filed a criminal complaint about this to the judicial authorities. In order not to disrupt the work of investigators, we will not give other details before the final conclusions.


We apologise for the incident. Thank you for your understanding.


Regards,


Octave
Click to expand...
Source: http://status.ovh.net/?do=details&id=5070&PHPSESSID=d2344fbaf05bddbe375071d4ec197f41
 
T

threz

New Member
Wow, that's actually pretty serious. Someone could have had SSH access to servers in Quebec for "a few days." That's... not good, to say the least. 
 
Aldryic C'boas

Aldryic C'boas

The Pony
Why hasn't this been sent out via e-mail to all their customers?
Click to expand...
Depending on when that was posted - it's possible they may be doing so now. It's worth keeping in mind that whenever I do a mass email, it takes over 6 hours to get them all sent - and we only have ~13k clients. No telling how large of a userbase they're having to send mail out to.

Or they've already sent out notifications, but thanks to how dirty their IP space is the emails are all getting spam binned :p
 
peterw

peterw

New Member
vld said:
Therefore, where the client has not removed our SSH key and has not changed their root password, we immediately changed the password of the servers in the BHS DC to cancel this risk there.
Click to expand...
So the root password was saved too?

threz said:
That's... not good, to say the least.
Click to expand...
That's a worst case scenario.
 
Last edited by a moderator:
R

rds100

New Member
Verified Provider
What about copies of the scanned IDs sent to OVH, were these stored? Were they stolen too?
 
scv

scv

Massive Nerd
Verified Provider
kaniini said:
It seems so.  My question is why were they retaining this information on the customer server?  Makes no sense to me.  :(
Click to expand...
Honestly, this seems like common practice with a lot of dedicated providers. Their less tech-savvy customers may not even login to the machine or even know how to. First step you should take when setting up a dedi is to remove any sort of access from the provider for this exact reason.
 
jarland

jarland

The ocean is digital
Well if anyone wants my boy meets world episodes and Dropbox full of church event clips... I hope they enjoy my OVH server.
 
Last edited by a moderator:
kaniini

kaniini

Beware the bunny-rabbit!
Verified Provider
scv said:
Honestly, this seems like common practice with a lot of dedicated providers. Their less tech-savvy customers may not even login to the machine or even know how to. First step you should take when setting up a dedi is to remove any sort of access from the provider for this exact reason.
Click to expand...
Well, I am talking about the initial password being stored in /root/.p.  But, all of the dedicated servers I have ever gotten, I just did my own OS install using IP-KVM or IPMI, much like my colo'd gear.
 
kaniini

kaniini

Beware the bunny-rabbit!
Verified Provider
Speck said:
Nobody should be using hashing for passwords anymore.
Click to expand...
Well, they use SHA512-CRYPT, as far as I understand.  And their status site update corroborates that.

The encryption password is "Salted" and based on SHA-512, to avoid brute-force attacks. It takes a lot of technical means to find the word password clearly. But it is possible.
Click to expand...
Of course, if they only do one round of SHA512-CRYPT, then it is still pretty trivial (a day or two per account) to crack the password I guess.

In my opinion they should reset passwords upon login and mail them to the contact e-mail.
 
  • Like
Reactions: acd
acd

acd

New Member
acd hurp-durp said:
kaniini said what I wanted to say. 2^N-round sha512 w/ resalting does not seem particularly better or worse than bcrypt. If you're going to post a link to support your post, could you instead post one where it explains the argument/reasoning/etc? A twitter post reiterating what you said does not a convincing argument make. I would really like to know why you think key derivation functions are better for password storage and while I know some theory, it's not hardly enough to derive it on my own.
Click to expand...
(edit: That algorithm is also known as PBKDF2 using HMAC-SHA512 as the PRF and password of salt||password. You got me with "using hashing for passwords" when you meant "using only hashing". And some days I lack the terminology to understand what is said.)
On topic, it sucks that OVH was compromised. I actually ticketed them yesterday morning about an unexplained reboot (nothing in my logs) of my BHS server on Friday; OVH told me their logs don't have anything regarding a reboot. Considering it's just a host node running sshd and half a dozen KVMs, I thought it was kind of weird. I wasn't using their supplied passwords, had no ovh ssh keys installed and there wasn't any unsecured & sensitive data on there. The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.
 
Last edited by a moderator:
wlanboy

wlanboy

Content Contributer
acd said:
The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.
Click to expand...
That is the worst part of it. A lot of server owners loose their sureness that their servers were not altered/touched.
 
kaniini

kaniini

Beware the bunny-rabbit!
Verified Provider
acd said:
(edit: That algorithm is also known as PBKDF2 using HMAC-SHA512 as the PRF and password of salt||password. You got me with "using hashing for passwords" when you meant "using only hashing". And some days I lack the terminology to understand what is said.)


On topic, it sucks that OVH was compromised. I actually ticketed them yesterday morning about an unexplained reboot (nothing in my logs) of my BHS server on Friday; OVH told me their logs don't have anything regarding a reboot. Considering it's just a host node running sshd and half a dozen KVMs, I thought it was kind of weird. I wasn't using their supplied passwords, had no ovh ssh keys installed and there wasn't any unsecured & sensitive data on there. The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.
Click to expand...
This was actually likely an overheat event.  OVH nodes restart on MCEs such as PROCHOT.

In BHS, all nodes are IPMI-based, so if you use ipmitool on the node, you can check the System Event Log for an MCE.
 
Last edited by a moderator:
You must log in or register to reply here.
Top
amuck-landowner