# Security incident at OVH



## vld (Jul 22, 2013)

> Hello,
> 
> 
> A few days ago, we discovered that the internal security of our offices in Roubaix had been compromised. After internal investigations we found that a hacker was able to gain access to an email account of one of our system administrators. With this email access, they was able to gain access to the internal VPN of another employee. Then with this VPN access, they was able to compromise the access of one of the system admins who deals with the internal backoffice.
> ...


Source: http://status.ovh.net/?do=details&id=5070&PHPSESSID=d2344fbaf05bddbe375071d4ec197f41


----------



## Quexis (Jul 22, 2013)

> The encryption password is based on SHA512


Nobody should be using hashing for passwords anymore.


----------



## threz (Jul 22, 2013)

Wow, that's actually pretty serious. Someone could have had SSH access to servers in Quebec for "a few days." That's... not good, to say the least.


----------



## MannDude (Jul 22, 2013)

Yikes. Pretty serious indeed.

/subscribed.


----------



## WelltodoInformalCattle (Jul 22, 2013)

Why hasn't this been sent out via e-mail to all their customers?


----------



## Aldryic C'boas (Jul 22, 2013)

> Why hasn't this been sent out via e-mail to all their customers?


Depending on when that was posted - it's possible they may be doing so now. It's worth keeping in mind that whenever I do a mass email, it takes over 6 hours to get them all sent - and we only have ~13k clients. No telling how large of a userbase they're having to send mail out to.

Or they've already sent out notifications, but thanks to how dirty their IP space is the emails are all getting spam binned


----------



## peterw (Jul 22, 2013)

vld said:


> Therefore, where the client has not removed our SSH key and has not changed their root password, we immediately changed the password of the servers in the BHS DC to cancel this risk there.


So the root password was saved too?



threz said:


> That's... not good, to say the least.


That's a worst case scenario.


----------



## WelltodoInformalCattle (Jul 22, 2013)

Forget the moronic question, completely forgot about that.


----------



## rds100 (Jul 22, 2013)

What about copies of the scanned IDs sent to OVH, were these stored? Were they stolen too?


----------



## kaniini (Jul 22, 2013)

peterw said:


> So the root password was saved too?


It seems so.  My question is why were they retaining this information on the customer server?  Makes no sense to me.


----------



## scv (Jul 22, 2013)

kaniini said:


> It seems so.  My question is why were they retaining this information on the customer server?  Makes no sense to me.


Honestly, this seems like common practice with a lot of dedicated providers. Their less tech-savvy customers may not even login to the machine or even know how to. First step you should take when setting up a dedi is to remove any sort of access from the provider for this exact reason.


----------



## jarland (Jul 22, 2013)

Well if anyone wants my boy meets world episodes and Dropbox full of church event clips... I hope they enjoy my OVH server.


----------



## MannDude (Jul 22, 2013)

rds100 said:


> What about copies of the scanned IDs sent to OVH, were these stored? Were they stolen too?


Valid concern, didn't think about that.


----------



## kaniini (Jul 22, 2013)

scv said:


> Honestly, this seems like common practice with a lot of dedicated providers. Their less tech-savvy customers may not even login to the machine or even know how to. First step you should take when setting up a dedi is to remove any sort of access from the provider for this exact reason.


Well, I am talking about the initial password being stored in /root/.p.  But, all of the dedicated servers I have ever gotten, I just did my own OS install using IP-KVM or IPMI, much like my colo'd gear.


----------



## kaniini (Jul 22, 2013)

Speck said:


> Nobody should be using hashing for passwords anymore.


Well, they use SHA512-CRYPT, as far as I understand.  And their status site update corroborates that.



> The encryption password is "Salted" and based on SHA-512, to avoid brute-force attacks. It takes a lot of technical means to find the word password clearly. But it is possible.


Of course, if they only do one round of SHA512-CRYPT, then it is still pretty trivial (a day or two per account) to crack the password I guess.

In my opinion they should reset passwords upon login and mail them to the contact e-mail.


----------



## BlackoutIsHere (Jul 22, 2013)

kaniini said:


> In my opinion they should reset passwords upon login and mail them to the contact e-mail.


 Probably a good idea. Lots of customers are so clueless about security these days.


----------



## wlanboy (Jul 23, 2013)

Any news what was leaked?


----------



## acd (Jul 23, 2013)

acd hurp-durp said:


> kaniini said what I wanted to say. 2^N-round sha512 w/ resalting does not seem particularly better or worse than bcrypt. If you're going to post a link to support your post, could you instead post one where it explains the argument/reasoning/etc? A twitter post reiterating what you said does not a convincing argument make. I would really like to know why you think key derivation functions are better for password storage and while I know some theory, it's not hardly enough to derive it on my own.


(edit: That algorithm is also known as PBKDF2 using HMAC-SHA512 as the PRF and password of salt||password. You got me with "using hashing for passwords" when you meant "using only hashing". And some days I lack the terminology to understand what is said.)
On topic, it sucks that OVH was compromised. I actually ticketed them yesterday morning about an unexplained reboot (nothing in my logs) of my BHS server on Friday; OVH told me their logs don't have anything regarding a reboot. Considering it's just a host node running sshd and half a dozen KVMs, I thought it was kind of weird. I wasn't using their supplied passwords, had no ovh ssh keys installed and there wasn't any unsecured & sensitive data on there. The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.


----------



## wlanboy (Jul 23, 2013)

acd said:


> The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.


That is the worst part of it. A lot of server owners loose their sureness that their servers were not altered/touched.


----------



## kaniini (Jul 23, 2013)

acd said:


> (edit: That algorithm is also known as PBKDF2 using HMAC-SHA512 as the PRF and password of salt||password. You got me with "using hashing for passwords" when you meant "using only hashing". And some days I lack the terminology to understand what is said.)
> 
> 
> On topic, it sucks that OVH was compromised. I actually ticketed them yesterday morning about an unexplained reboot (nothing in my logs) of my BHS server on Friday; OVH told me their logs don't have anything regarding a reboot. Considering it's just a host node running sshd and half a dozen KVMs, I thought it was kind of weird. I wasn't using their supplied passwords, had no ovh ssh keys installed and there wasn't any unsecured & sensitive data on there. The incidents are probably unrelated and I'm overreacting but I might as well rebuild it anyway.


This was actually likely an overheat event.  OVH nodes restart on MCEs such as PROCHOT.

In BHS, all nodes are IPMI-based, so if you use ipmitool on the node, you can check the System Event Log for an MCE.


----------



## bfj (Jul 24, 2013)

kaniini said:


> In BHS, all nodes are IPMI-based, so if you use ipmitool on the node, you can check the System Event Log for an MCE.


Really? I just tried it on my OVH-S16 Dedicated and it threw: 

Setting up ipmitool (1.8.11-5) ...

[....] Starting IPMI event daemon ipmievdipmievd: using pidfile /var/run/ipmievd.pid0

Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

Unable to open interface

 failed!

 

Are you sure this is not a feature you have to add?


----------



## scv (Jul 24, 2013)

Probably needs the module loaded?


----------



## bfj (Jul 26, 2013)

scv said:


> Probably needs the module loaded?


Just tried, unless you have to being using the shit OS that OVH comes with default. 

st# modprobe ipmi_si

ERROR: could not insert 'ipmi_si': No such device

st# modprobe ipmi_kcs_drv

FATAL: Module ipmi_kcs_drv not found.

 

Still no go unfortunately.


----------



## scv (Jul 26, 2013)

Well, no such device means no such device. Looks like not every device is IPMI enabled. yes, I'm a member of the tautology club.


----------



## kaniini (Jul 28, 2013)

What does dmidecode say the motherboard is?


----------



## acd (Jul 31, 2013)

Probably the same as mine:


```
Manufacturer: Intel Corporation
Product Name: DH61AG
Version: AAG23736-503
```
No IPMI on there, unfortunately.


----------

