# Told my server is out of memory by the Datacenter techs?



## coreyman (Jun 25, 2013)

So I've never had this happen before. A DC tech opened a ticket with me and told me that 'Your machine 1638CC-XEON is beeping due to it being out of memory. Please check into this issue and advise.'
 
So I checked logs and all I could see is where some containers are running out of memory and openvz is killing some processes in their container like it should. I checked the disks and temp sensors in the server to make sure it wasn't 'beeping' for some other reason. Check out the ticket correspondence.
 
 
 
Them: Sierra, Your machine 1638CC-XEON is beeping due to it being out of memory. Please check into this issue and advise. Thanks,
 
Me: I've never heard of an out of memory beep. Naturally linux uses a lot of the RAM for disk cache but will free that up at anytime. (about 12GB used for cache). free -m total used free shared buffers cached Mem: 32014 31718 295 0 920 11857 -/+ buffers/cache: 18940 13073 Swap: 34815 3701 31114 So then I checked the disks with smartctl and they all appear to be good. Then I checked the temp sensors and they all seem good. (all under 31C) Can you confirm the 'out of memory beep' with a manual link or something?
 
Them: Sierra, Log into IPMI and you will see the Out of memory messages posted. Thanks,
 
Me: I am currently unable to login to IPMI on this machine but I did check dmesg and it looks like the last time there was an OOM from openvz was about 12 days ago. Is the machine beeping when there is DMESG errors being printed to console?
 
Them: Sierra, I have attached an image of the error message the device is throwing. Also are you using the public IPMI address as we are able to log into IPMI.
 
Me: Yes, it's just that I'm on linux right now. I'm also not seeing any 'attachments'.
 
Them: Sierra, Attachment are at the top of the ticket.
 
Me: Ok I read that, and it appears some containers on the node are running out of memory.... and OpenVZ is doing its job and killing processes in that container. Nothing to worry about. Can you please confirm that my server is beeping due to these dmesg logs being printed to the console?

Them: Sierra, Our on-site tech confirmed the beeping is due to the device running out of memory, how would you like us to proceed?
 
Me: I don't understand. The server is not out of memory.


----------



## shovenose (Jun 25, 2013)

that seems really bizarre. Why would RAM usage cause it to beep?


----------



## D. Strout (Jun 25, 2013)

What DC is this? Also, how much memory does the server have?


----------



## MartinD (Jun 25, 2013)

Tell them you _don't_ want them to proceed with anything and to leave it alone.


----------



## happel (Jun 25, 2013)

What a nice feature, my computer only beeps when it has no memory .


----------



## JDiggity (Jun 25, 2013)

happel said:


> What a nice feature, my computer only beeps when it has no memory .


In any of the slots!

Disable beeps in the bios?


----------



## coreyman (Jun 25, 2013)

D. Strout said:


> What DC is this? Also, how much memory does the server have?


Not naming any names because I don't want to bring shame to anyone at this time. The server has 32GB of RAM.



shovenose said:


> that seems really bizarre. Why would RAM usage cause it to beep?


I thought so too.



happel said:


> What a nice feature, my computer only beeps when it has no memory .


Yea I've never had one beep when it was out of RAM.



MartinD said:


> Tell them you _don't_ want them to proceed with anything and to leave it alone.


What are they trying to do here? I'm really confused as to why they opened the ticket with me in the first place.


----------



## JDiggity (Jun 25, 2013)

coreyman said:


> What are they trying to do here? I'm really confused as to why they opened the ticket with me in the first place.


Maybe the beeping annoyed the tech?  Can't you turn off certain alarms in the Bios?


----------



## wdq (Jun 25, 2013)

24khost said:


> Maybe the beeping annoyed the tech?  Can't you turn off certain alarms in the Bios?



I'm sure they could just open up the server and unplug the cable that connects the beeper to the motherboard. Wouldn't even require a reboot.


----------



## coreyman (Jun 25, 2013)

24khost said:


> Maybe the beeping annoyed the tech?  Can't you turn off certain alarms in the Bios?


I'm sure you can but going into BIOS would require a ton of customers being down for whatever period of time it takes to find this beep setting. I'm just renting this server, they could have turned off the beeping before they sold it to me.


----------



## coreyman (Jun 25, 2013)

wdq said:


> I'm sure they could just open up the server and unplug the cable that connects the beeper to the motherboard.


Some mobos have the beeper soldered on to it.


----------



## JDiggity (Jun 25, 2013)

coreyman said:


> Some mobos have the beeper soldered on to it.


Most newer ones.


----------



## D. Strout (Jun 25, 2013)

Beeper soldered on to the motherboard or not, I've never heard of _any_ computer beeping because it's out of memory. Usually it just silently starts swapping. I'm as confused as *@**coreyman* as to why the DC techs would say this is an out of memory problem.


----------



## MartinD (Jun 25, 2013)

It wont be that, it'll be because output is being piped to the console and that causes the beeps.


----------



## coreyman (Jun 25, 2013)

MartinD said:


> It wont be that, it'll be because output is being piped to the console and that causes the beeps.


That's what I asked them and they still want to say 'its because its out of memory'.......


----------



## Aldryic C'boas (Jun 25, 2013)

setterm -blength 0

Disabled audible bells (more specifically, sets any audible alert to a 0 length).  You would have to routinely watch your heat after running this, however.


----------



## Maximum_VPS (Jun 25, 2013)

Never heard of a motherboard beeping because of no memory. And i see some have mentioned that its beeping due to console output, but i would wonder about cpu overheating?


----------



## MartinD (Jun 25, 2013)

Aldryic C said:


> setterm -blength 0
> 
> Disabled audible bells (more specifically, sets any audible alert to a 0 length).  You would have to routinely watch your heat after running this, however.


^ This.


----------



## acd (Jun 25, 2013)

should that be



setterm -blength 0 > /dev/console
or does -blength affect all terminals? I thought setterm was local session only.


----------



## mikho (Jun 25, 2013)

Yeah, my guess the beeps are because the message is posted on the terminal. Not the out of memory thing itself. It could have been any message at all.


----------



## Wintereise (Jun 25, 2013)

That naming convention is distinct of ColoCrossing, just sayin'.


----------



## Jack (Jun 25, 2013)

D. Strout said:


> What DC is this? Also, how much memory does the server have?


_*'Your machine 1638CC-XEON*_

Not obvious? Look after the ID before -XEON


----------



## AnthonySmith (Jun 25, 2013)

Obvious reply would be: Your terms say nothing about beeping, I will allow my server to beep as much as I beeping want, what the beep has it got to do with you


----------



## Jack (Jun 25, 2013)

AnthonySmith said:


> Obvious reply would be: Your terms say nothing about beeping, I will allow my server to beep as much as I beeping want, what the beep has it got to do with you


Would be quite annoying though?


----------



## Wintereise (Jun 25, 2013)

AnthonySmith said:


> Obvious reply would be: Your terms say nothing about beeping, I will allow my server to beep as much as I beeping want, what the beep has it got to do with you


Colocation, while a shame, does not work like that. Even if this is a dedi, same terms apply.

You'll be disturbing people while they try to work / etc, which pretty much is valid grounds for at least attempting to boot you.


----------



## D. Strout (Jun 25, 2013)

Jack said:


> Not obvious? Look after the ID before -XEON


Actually not correct ^_^  Presumably the memory was upgraded. It is displayed in the post in the free -m output though, so that is my bad.


----------



## Jack (Jun 25, 2013)

D. Strout said:


> Actually not correct ^_^  Presumably the memory was upgraded. It is displayed in the post in the free -m output though, so that is my bad.


What are you on about?

I know CC's Labels are SERVERID-CC for Colocrossing then -XEON FOR XEON's, C2Q for Core2Quad , AMD for AMD's...


----------



## D. Strout (Jun 25, 2013)

I thought you were referring to the other part of my post asking how much memory, since the ID had "1638" in it. As in, 16,384MB of memory, or 16GB, when the free -m output said 32,768MB of memory. I get it now.


----------



## kaniini (Jun 25, 2013)

It could be beeping due to a memory error -- i.e. one of the chips is throwing ECC errors; some of my dells do that (they also light the attention light amber).  They also display a message on the LCD panel which typically shows the serial number saying which bank is bad.

That said, how could you hear a server beeper in a properly cooled DC -- surely the ventilation noise would drown it out.

At least in any datacenter _I_ have been in (barring FDC) this was the case.


----------



## rsk (Jun 25, 2013)

kaniini said:


> That said, how could you hear a server beeper in a properly cooled DC -- surely the ventilation noise would drown it out.


I think at CC they have nothing else better to do, than to listen to beeps


----------



## Jack (Jun 25, 2013)

rsk said:


> I think at CC they have nothing else better to do, than to listen to beeps


Na, they are busy helping CVPS..


----------



## Zen (Jun 25, 2013)

This makes no sense, not the beeping part, but the fact that they hear or care about it. In any proper DC or filled up rack.. I doubt you would hear the beep to the point where it was frustrating you.


----------



## mikho (Jun 25, 2013)

Zen said:


> This makes no sense, not the beeping part, but the fact that they hear or care about it. In any proper DC or filled up rack.. I doubt you would hear the beep to the point where it was frustrating you.


Unless you are working very close to the rack, I would look into a beeping noise if it was me working there.


I would be thankful if a tech opened a ticket like that for me to investigate.


----------



## Zen (Jun 25, 2013)

mikho said:


> Unless you are working very close to the rack, I would look into a beeping noise if it was me working there.
> 
> 
> I would be thankful if a tech opened a ticket like that for me to investigate.


Eh, with proper cooling and 40+ servers running.. I don't know.


----------



## kaniini (Jun 25, 2013)

Zen said:


> Eh, with proper cooling and 40+ servers running.. I don't know.


I've had to diagnose beepcodes in a cabinet before, I had to get my ear very close to the server to hear anything.

So yeah, it makes me wonder about the cooling at CC, for sure.


----------



## mikho (Jun 25, 2013)

Zen said:


> Eh, with proper cooling and 40+ servers running.. I don't know.


It's hard to hear, I admit that but once you hear that first beep you go wtf and shake it of, when the second comes you wtf again and some people (like me) wont stop until they find the source.


----------



## scv (Jun 26, 2013)

Wintereise said:


> Colocation, while a shame, does not work like that. Even if this is a dedi, same terms apply.
> 
> You'll be disturbing people while they try to work / etc, which pretty much is valid grounds for at least attempting to boot you.


I don't think most DCs actually care about this though. The rack next to ours in Detroit has had an Adaptec screaming away the entire time we've been in the facility.


----------



## Marc M. (Jun 26, 2013)

Get yourself a bottle of Premium RAM Fluid    to top it off when it starts beeping


----------



## earl (Jun 26, 2013)

Did you ask the DC what the beep code was? check it with your board manufacturer.. I'm sure it's something hardware related or maybe a temperature sensor was set off..


----------



## JDiggity (Jun 26, 2013)

They probably found the beep while working on another persons server that was right next to it.  When they went to derack it to work on it.


----------



## TheLinuxBug (Jun 26, 2013)

kaniini said:


> I've had to diagnose beepcodes in a cabinet before, I had to get my ear very close to the server to hear anything.


 

A good friend of mine once gave me a link to what he considered "Data Center Tech Must Haves" and included on the list was a Stethoscope for trouble shooting this exact thing.  In most cases, as stated above, if properly cooled, you shouldn't be hearing the audible beeps without purposely listening for them (or being very close to the server for some reason).

Cheers!


----------



## HalfEatenPie (Jun 26, 2013)

*@**TheLinuxBug*, you should totally share the link if you can find it.  I'd be interested in it.


----------



## Wintereise (Jun 27, 2013)

scv said:


> I don't think most DCs actually care about this though. The rack next to ours in Detroit has had an Adaptec screaming away the entire time we've been in the facility.


Both Equinix and Internap do not take kindly to random sounds, that I can tell from experience.

No idea about others though, but it should be inaudible unless you're really close either way.


----------



## coreyman (Jun 27, 2013)

I checked temperature sensors... I think the server is beeping from console output.


----------



## AlexBarakov (Jun 27, 2013)

If the server is beeping, there is definitely something wrong. Disabling the beepings is ignoring the problem. Diagnose, find and resolve it. Might be a failing sensor of something. Ask for further information, the beep code and possibly take the machine offline if needed. Server's don't always beep for no reason and be glad that someone from the DC caught it.

Also what is the information you get from IPMI?


----------



## sleddog (Jun 27, 2013)

Alex_LiquidHost said:


> Server's don't always beep for no reason


Maybe it's backing up :lol:


----------



## Aldryic C'boas (Jun 27, 2013)

sleddog said:


> Maybe it's backing up :lol:


That was painful <_<


----------



## coreyman (Jun 27, 2013)

Alex_LiquidHost said:


> If the server is beeping, there is definitely something wrong. Disabling the beepings is ignoring the problem. Diagnose, find and resolve it. Might be a failing sensor of something. Ask for further information, the beep code and possibly take the machine offline if needed. Server's don't always beep for no reason and be glad that someone from the DC caught it.
> 
> Also what is the information you get from IPMI?



It's attached to the intial post.


----------

