# heads up: kernel regression in CentOS 2.6.32 and OpenVZ 2.6.18/2.6.32 kernels creates local DoS (reb



## kaniini (Jun 30, 2013)

Rack911 posted a vulnerability to oss-security earlier this evening.

[temp removed]

We (my company) did some of the low-level debugging work on this vulnerability and developed an exploit which reliably triggered it.  It is a race condition related to RCU of kernel data structures in the IPv4 stack.

We decided to release it publically because there was already POC code from a similar bug from 2003 that had about a 50% chance of whacking it anyway.

CVE number will be forthcoming.


----------



## Steven (Jun 30, 2013)

Well I guess this made it here. All started with us trying to break betterlinux, after we discovered a exploit from 2003 caused some odd behavior we had kaniini take a look and he made it a fully reproducible poc.


----------



## Francisco (Jun 30, 2013)

Tested it against latest .18 and latest dev .32's.

Annoying 

Francisco


----------



## rds100 (Jun 30, 2013)

Damn, i hate reboots for kernel upgrades. Ant the customers hate it too.


----------



## kaniini (Jun 30, 2013)

Fix: http://turtle.dereferenced.org/~nenolod/hemlock-fix.patch

I am sure someone can build an RPM with this in it.  Personally, dealing with rpmbuild is something I would rather not do.


----------



## maounique (Jun 30, 2013)

So this is probably related with our nodes reboots for ovz. If this fixes it, then we can restart selling those 

I always suspected there is some exploit and someone is running it randomly on a handful of nodes (very old nodes are not affected).

Also, oddly, after we took out the stock and decided not to sell ovz anymore, these reboots halved or so.


----------



## Nick_A (Jun 30, 2013)

Yeah rebooting again right now would be rough.


----------



## SeriesN (Jun 30, 2013)

Dangit! Discovered it this weekend and was planning to update by next sunday but it is public now .


----------



## Francisco (Jun 30, 2013)

SeriesN said:


> Dangit! Discovered it this weekend and was planning to update by next sunday but it is public now .


Been trying to get a ksplice to generate but it's being annoying. I've yet to build a 2.6.32 ksplice so I'm running into a few derps for sure.

Francisco


----------



## SeriesN (Jun 30, 2013)

Francisco said:


> Been trying to get a ksplice to generate but it's being annoying. I've yet to build a 2.6.32 ksplice so I'm running into a few derps for sure.
> 
> 
> Francisco


Sir,

You HAZ Ksplice? @[email protected]


----------



## Francisco (Jun 30, 2013)

SeriesN said:


> Sir,
> 
> You HAZ Ksplice? @[email protected]


Not on a paid subscription. We roll our own to patch things when we don't want to reboot 90 nodes. It's easier to distro a 100k patch than reboot everyone 

To date we've had to roll a half dozen or so ksplices, usually to fix deadlocks. Ksplice is giving me attitude over .32, though, so we'll see.

Francisco


----------



## SeriesN (Jun 30, 2013)

Francisco said:


> Not on a paid subscription. We roll our own to patch things when we don't want to reboot 90 nodes. It's easier to distro a 100k patch reboot everyone
> 
> 
> To date we've had to roll a half dozen or so ksplices, usually to fix deadlocks. Ksplice is giving me attitude over .32, though, so we'll see.
> ...



Are you willing to "license" out your Ksplice and patches? I am sure some unfortunate soul would appreciate that.


----------



## Francisco (Jun 30, 2013)

SeriesN said:


> Are you willing to "license" out your Ksplice and patches? I am sure some unfortunate soul would appreciate that.


Let me get this damn thing to compile properly before I start giving things away 

Charging for opensource things is wrong, especially since kaniini wrote the patch my builds based from.

Francisco


----------



## SeriesN (Jun 30, 2013)

Francisco said:


> Let me get this damn thing to compile properly before I start giving things away
> 
> 
> Charging for opensource things is wrong, especially since kaniini wrote the patch my builds based from.
> ...



Well, people charge for "PHP" and "linux" related works. Just saying


----------



## Francisco (Jun 30, 2013)

SeriesN said:


> Well, people charge for "PHP" and "linux" related works. Just saying


The linux people charge for support  Anyways that's another subject all together.

Right now i've patched against 76.8 and just waiting for it to chew through. I've given the thing an E3 to run on but it seems it isn't forking multiple threads like I was hoping.

Francisco


----------



## Aldryic C'boas (Jun 30, 2013)

We're in the business of selling VPSes - why does everyone keep assuming we exist to do their coding work for them?


----------



## SeriesN (Jun 30, 2013)

Aldryic C said:


> We're in the business of selling VPSes - why does everyone keep assuming we exist to do their coding work for them?


Cause You are GOOOOOD at it. Good morning boss, grab a coke and cheer up


----------



## Aldryic C'boas (Jun 30, 2013)

Doesn't mean we work for free.. and no amount of caffeine or liquor is going to make me start believing that we should function as a free, public helpdesk <_<


----------



## SeriesN (Jun 30, 2013)

SeriesN said:


> Are you willing to "license" out your Ksplice and patches? I am sure some unfortunate soul would appreciate that


 





Aldryic C said:


> Doesn't mean we work for free.. and no amount of caffeine or liquor is going to make me start believing that we should function as a free, public helpdesk



Me Tell before but Big Unicorn Said NOOOOOOOOOOOOOOOOOOOOOOOOOOOO


----------



## Magiobiwan (Jun 30, 2013)

Is the 2.6.18 kernel line affected? Namely the latest stable version kernel.


----------



## jarland (Jun 30, 2013)

Aldryic C said:


> Doesn't mean we work for free.. and no amount of caffeine or liquor is going to make me start believing that we should function as a free, public helpdesk <_<


Invoice me for every time I've asked Fran for advice and I'll pay that with a smile 

People should be paid for what they do, Fran just does so much awesomeness that the list would probably crash WHMCS


----------



## Magiobiwan (Jun 30, 2013)

And the PayPal fees would be outrageous!


----------



## Francisco (Jun 30, 2013)

Magiobiwan said:


> Is the 2.6.18 kernel line affected? Namely the latest stable version kernel.


Yus.

RHEL 2.6.32 is a complete cluster fuck since it has so much crap backported. I simply can't get a ksplice to generate properly. I had to change how some of the structs were handled

Right when I *think* I have it figured out? it blows up again.

Francisco


----------



## kaniini (Jun 30, 2013)

RH still claims they can't reproduce this.  Whatever.

This is CVE-2013-2224


----------



## kaniini (Jun 30, 2013)

Francisco said:


> RHEL 2.6.32 is a complete cluster fuck since it has so much crap backported.


I wasn't able to get hemlock.c to run successfully on Debian 6, which also uses 2.6.32.  This leads me to believe the vulnerability is specific to RHEL's kernel series.  I bet some of their modifications opened the possibility of this vulnerability.


----------



## Francisco (Jun 30, 2013)

kaniini said:


> I wasn't able to get hemlock.c to run successfully on Debian 6, which also uses 2.6.32.  This leads me to believe the vulnerability is specific to RHEL's kernel series.  I bet some of their modifications opened the possibility of this vulnerability.


If you have a bunch of kernels to try, try something from the 2.6.39 range.

A long time ago I had a developer work on a fairly large addition to the .32 kernels and he went on a huge rant about how RHEL's .32 is pretty much .39.

Francisco


----------



## kaniini (Jun 30, 2013)

I'll build a 2.6.39 kernel and try it out in a bit if I can spare some free time.


----------



## Steven (Jul 1, 2013)

Redhat Bugzilla

https://bugzilla.redhat.com/show_bug.cgi?id=979936


----------



## KuJoe (Jul 1, 2013)

OpenVZ.org released a new kernel today to address it.


----------



## eva2000 (Jul 1, 2013)

Francisco said:


> If you have a bunch of kernels to try, try something from the 2.6.39 range.
> 
> 
> A long time ago I had a developer work on a fairly large addition to the .32 kernels and he went on a huge rant about how RHEL's .32 is pretty much .39.
> ...


Has anyone tried CentOS with Oracle Linux 6 UEK 2.6.39 kernel (3.0.16 mainline) ?


----------



## kaniini (Jul 2, 2013)

RedHat has confirmed it is an RHEL-specific regression caused by a badly done rebase.


----------



## rds100 (Jul 2, 2013)

@kaniini so they were able to reproduce it finaly?


----------



## concerto49 (Jul 2, 2013)

rds100 said:


> @kaniini so they were able to reproduce it finaly?


Or more like finally willing to acknowledge it.


----------



## NodeBytes (Jul 2, 2013)

removed.


----------

