Tool for detecting if a VPS node is "slabbed" or not

[kaniini]

Based on the GVH new Dallas node running in a hypervisor, I decided to begin writing a tool to probe an OpenVZ/LXC environment and see if it is slabbed.

There's a few ways we can do this.  First of all, almost all hypervisors provide basic information at CPUID leaf 0x40000000.  So, we can sniff that and see if there's a hypervisor running.  This is basically essential, because they all use CPUID to learn more about the VMM in early boot... this allows them to learn how to make hypercalls and so on.

Once we know more about the basic hypervisor... we can do more advanced things.  For example, it would maybe be possible to bitbang on VMware's bus by using inl() and outl() calls in combination with ioperm().  That would allow us to learn more about the hypervisor (such as version), etc.

Code here: https://github.com/kaniini/slabbed-or-not

It'd be interesting if people ran it, especially on machines they know not to be slabbed, so I can verify that the code does the right thing on a baremetal environment.

 

[drmike]

Bad ass... so for a non git head like me...  Explain how to git this and all that jazz... so I can automate it and include in my standard VPS pre-use testing....

 

[kaniini]

Just download https://github.com/kaniini/slabbed-or-not/archive/0.1.tar.gz

Then... cd slabbed-or-not-0.1; make; ./slabbed-or-not

 

[drmike]

Well that's simple... 

Funny, GVH-Jon / GreenValueHost offer created the motivation for the slab-detector...  This is going to be all the rage once other folks notice it 

The horrors as people try to mask details...

 

[drmike]

First go, I got this:

root@ohcanada:~/slabbed-or-not-0.1# ./slabbed-or-not
Illegal instruction

 

[drmike]

Then another VPS..... OpenVZ on this one...

~/slabbed-or-not-0.1# make; ./slabbed-or-not
gcc -o slabbed-or-not slabbed-or-not.c
Illegal instruction

 

[kaniini]

Try version 0.1.1.  It had a late-breaking fix.

Basically, to detect if we're running on Xen PV or Xen HVM, we pass an illegal instruction.  On Xen PV, it will be interpreted as a normal instruction, on HVM it will trigger SIGILL (but that doesn't matter because the initial check will pass).

There's a few things we can do with VMware too like this, but I haven't bothered yet.

edit: https://github.com/kaniini/slabbed-or-not/archive/0.1.1.tar.gz is 0.1.1 if anyone is confused.

 

[drmike]

Well that indeed works...   Good work.

 

[Damian]

Anyone else feel that this is a case of "OMFG WE'VE FOUND A PROBLEM THAT WE MADE OURSELVES LET'S HAVE A HUGE DEBACLE ABOUT IT" for something that's not actually a problem? It's not like anyone's running on single-core Xeons from 2004; if the server can handle nested virt, what's the big deal?

We don't do this ourselves, so I think I might be missing the point. I'm open to education, of course.

 

[drmike]

Well, we had a thread around here where some providers volunteered that they use slabs and gave good reasons why...  Recommended reading for those who haven't disclosed using such, if they need more logic conversationally speaking.

I'm sure we all know providers using such for good and others for wrong/abusive reasons.  I'm not even opening my mouth on this one...  May the furniture fly and land where it may and best of luck to all fingerbanged by this script.  opcorn:

 

[kaniini]

Anyone else feel that this is a case of "OMFG WE'VE FOUND A PROBLEM THAT WE MADE OURSELVES LET'S HAVE A HUGE DEBACLE ABOUT IT" for something that's not actually a problem? It's not like anyone's running on single-core Xeons from 2004; if the server can handle nested virt, what's the big deal?

We don't do this ourselves, so I think I might be missing the point. I'm open to education, of course.

Well, really, I don't care if people use slabs at all... I think you misinterpret my reasoning for releasing this tool.

There are some fairly legitimate use cases where slabbing might be advantageous -- live migrations being one of them.  If you've already put in high-availability infrastructure, it is a way where you can kind of get the advantages of HA for your OpenVZ deployments.  That's fine, really.

But then you have the dodgier hosts which do slabbing solely as a way to overcommit their servers even further (by taking advantage of slabbing to split a physical server up into multiple scheduling domains) -- these are the ones which are less than honest about their use of hypervisors on their OpenVZ deployments.

That said: I do believe that if you are slabbing, then you should be forthright about it.  Then you have nothing to hide, and nothing to fear from this tool, right?

edit: I mean, basically, the only reason why I wrote this was because I bought a VPS last night that was so awful, I actually felt compelled to investigate why.  I mean, there was a lot of things that seemed off about it -- CPU info was whack, hypervisor bit was set, performance was crap.  Of course any reasonable person is going to investigate what hypervisor their container is running under if they see these things.  And, it's actually important that this tool exists, because it can explain strange behaviour with a VPS -- think about steal-time for example... that's something that you cannot see inside OpenVZ, and which only exists under a hypervisor.

 

[Damian]

Good response, I do fully understand it now. The original posts had a "LOL WE'VE GOT THEM NOW" feel to it.

 

[DomainBop]

Funny, GVH-Jon / GreenValueHost offer created the motivation for the slab-detector..

Slabbing would explain how he's going to use that new /24 that's SWIPed to him...  Phoenix is apparently next on the GVH slabathon with another /24 there http://whois.arin.net/rest/org/GVH-8/pft

edit:

I mean, basically, the only reason why I wrote this was because I bought a VPS last night that was so awful, I actually felt compelled to investigate why.  I mean, there was a lot of things that seemed off about it -- CPU info was whack, hypervisor bit was set, performance was crap.

Ginernet was slabbing (openvz inside kvm, 2 kvm's per server) and there was a noticeable performance hit in disk performance of 50%+ with the SSD drives they were using.  I think they stopped slabbing now but they were one of the few hosts who openly admitted to doing it (the ease of live migrations you mentioned was one reason they did it).

In terms of stability that slabbed VPS was a nightmare with frequent reboots but that may have been due to the node and the data centers both being DDoS magnets rather than the slabbing (although KVM's tendency to lock up when it gets hit with a DDoS may have been a contributing factor to the reboots).

 

[kaniini]

Good response, I do fully understand it now. The original posts had a "LOL WE'VE GOT THEM NOW" feel to it.

Hmm... I am not sure why they would... my post with this thread was basically a technical explanation of how the thing works, as well as the motivation for why I would release a polished up tool for it.

 

[DomainBop]

There's a few things we can do with VMware too like this, but I haven't bothered yet.

The OVH Classic and Low Latency VPS lines are OpenVZ running on VMWare

 

[drmike]

"...reasonable person is going to investigate what hypervisor their container is running under if they see these things."

Maybe your "average" vpsBoard user does... but don't expect the peasants in sillyville to follow your high standards Lord kaniini.  For they, they starve from lack of knowledge.

Lips shut. Done.  Carry on.

 

[AuroraZero]

"...reasonable person is going to investigate what hypervisor their container is running under if they see these things."

Maybe your "average" vpsBoard user does... but don't expect the peasants in sillyville to follow your high standards Lord kaniini.  For they, they starve from lack of knowledge.

Lips shut. Done.  Carry on.

Do not worry man I make this same mistake all the time as well. I assume just because I, or in this case we, do it everyone else does the same. When the truth is the average person does not do it nor do they have same level of knowledge in the given field even.

 

[Rallias]

Ginernet was slabbing (openvz inside kvm, 2 kvm's per server) and there was a noticeable performance hit in disk performance of 50%+ with the SSD drives they were using.

I don't see that kind of IO degradation.



In fact, on nodes that I've seen that have been slabbed properly (xen pv, elevator=noop, openvswitch bridge), I've seen improved performance.

 

[concerto49]

I don't see that kind of IO degradation.



In fact, on nodes that I've seen that have been slabbed properly (xen pv, elevator=noop, openvswitch bridge), I've seen improved performance.

That's the thing. Why does it matter? What's the point here but drama? Does it work? Does it perform? Is it reliable? I think that's what matters to end users. They want a working product that's performant. They want the features.

 

[kaniini]

That's the thing. Why does it matter? What's the point here but drama? Does it work? Does it perform? Is it reliable? I think that's what matters to end users. They want a working product that's performant. They want the features.

No drama intended... it's just a tool designed to answer the question of whether or not you're running in a hypervisor or not.

Isn't the question "can you determine whether you are running in a hypervisor from within a restricted container" interesting enough without a drama angle?  If not that, then is "can we determine as much information about the hypervisor from inside the container" interesting?