Mass drive failings linked to accidental gas (fire suppression) release in Sydney data centre

[Oliver]

Came across this interesting thread on the AUSNOG mailing lists today. I don't have any equipment in this datacentre myself so am not affected but there must be plenty of unhappy people around after this event:

http://lists.ausnog.net/pipermail/ausnog/2014-January/thread.html#22079
http://lists.ausnog.net/pipermail/ausnog/2014-January/022079.html
 

Hi All,
I have quite a few people today with hard drives fail in suites on
level 4 at global switch. GS tell us the Gas was dropped in response to
some smoke from some plant. Not sure how inergen would effect hard
drives but there are many people wondering around on level 4 looking
like there dog just died.

Anybody have any more info.

Matt

Appears a gas fire suppression system release caused a noise or air pressure change that caused many hard drives in the facility to fail; with certain drives and servers being generally more vulnerable.

Anyone seen something like this occur before? Might be more of a concern for LEB providers who are using cheaper SATA drives if you are in a facility with this type of system and suddenly have a huge number of drives to replace. Some replies have indicated that the system in use is not the same as in other competing datacentres in the same city (Equinix for example, where my own business has some equipment).

Granted it's not something that happens often but when it does if you have to replace a bunch of drives at once could certainly be quite frustrating!

Related links:

Inert Gas Data Center Fire Protection and Hard Disk Drive Damage 

http://www.datacenterjournal.com/it/inert-gas-data-center-fire-protection-and-hard-disk-drive-damage/

Siemens white paper

http://www.hgi-fire.com/wp-content/uploads/2013/03/White-Paper-potential-problems-with-computer-hard-disks-V1.3.pdf

 

[blergh]

I have a feeling that someone is going to have to pay for that and it won't be the owners.

 

[rds100]

I don't understand how the gas can make the HDDs fail. The HDD platters are inside an air tight space. The gas cannot go there. Or was the gas corrosive and cause the HDD electronics board to be damaged? Doubtful, because then it wouldn't be just HDDs failing.

I think that more probably the ventilation / cooling was turned off and the HDDs overheated and this is the reason for so many failures.

 

[Oliver]

I don't understand how the gas can make the HDDs fail. The HDD platters are inside an air tight space. The gas cannot go there. Or was the gas corrosive and cause the HDD electronics board to be damaged? Doubtful, because then it wouldn't be just HDDs failing.

I think that more probably the ventilation / cooling was turned off and the HDDs overheated and this is the reason for so many failures.

I think it's the pressure change and or noise that is the problem not the gas entering the drives and causing electrical failures in the way you have described.

There are videos of people on YouTube yelling at big drive arrays and the noise vibrations do have an effect on disks that is quite easily visible with basic reporting tools. I don't think there was a cooling problem in the datacentre - it was just an accidental gas drop not a temperature issue.

 

[Damian]

Drives are sealed against dust and contaminates, not air pressure changes. The point of these...



... are to equalize the air pressure within the drive. The intended purpose to release pressure as the drive heats up and to prevent a vacuum as the drive cools down. As hard disk heads work on Bernoulli's principal to fly above the disk platter at a specific distance, sudden extreme changes in pressure will cause issues. 

Some basic Google searching specifies that yes, a pressurized gas fire suppression system in enclosed room will cause air pressure variations, but I can't really find any information on what kind of levels could be expected.

I found this image:



At these distances, it's not unlikely or unreasonable to believe that a large air pressure change that the drive couldn't react to quick enough that the head wouldn't have slammed into the platter.

This is, however, one of those freak occurrences that's hard to plan for. Even fancy datacenter certifications don't account for things like this.

 

[ndelaespada]

This is not the first time this happens... have heard of several cases like this, I even was at a DC once while they were testing this system and at the time of the testing several hds failed.

 

[tonyg]

See here:

http://www.youtube.com/watch?v=tDacjrSCeq4

Tony

 

[tchen]

Inergen's mix is pressure neutral.  Most reports point to it being damage from the alarm and discharge noise associated with the release.

From their FAQ https://www.ansul.com/en/us/DocMedia/F-2012095.pdf
 

[SIZE=11pt]High decibel levels with high frequency content have been found to disrupt HDD performance. [/SIZE]

[SIZE=11pt]The mere introduction of INERGEN agent into an enclosure containing hard disk drives does not result in degradation of HDD performance.[/SIZE]

[SIZE=11pt]Tyco Fire Protection Products (Tyco) has indentified installation techniques to minimize disru[/SIZE]ptive noise generated in a data center environment. For more information, please see our position paper on INERGEN Acoustics Testing (Form No. F-2012029). 

Edit: Okay, now I read through all the other links so acoustic noise is no surprise - so it's just me restating the obvious again.  Moving along

 

[bauhaus]

I don't understand how the gas can make the HDDs fail. The HDD platters are inside an air tight space. 

Drives are sealed against dust and contaminates, not air pressure changes.

Consumer and enterprise grade HDD's are not completely sealed. The hermetically sealed are used in space research, the military field and in extreme environments and are quite expensive. The closest to these hard drives is the new one manufactured by WD 6TB filled with helium and fully sealed. I'm not sure if these drives react in the same way to high level of acoustic noise.

 

[Magiobiwan]

Consumer and enterprise grade HDD's are not completely sealed. The hermetically sealed are used in space research, the military field and in extreme environments and are quite expensive. The closest to these hard drives is the new one manufactured by WD 6TB filled with helium and fully sealed. I'm not sure if these drives react in the same way to high level of acoustic noise.

They likely would. The noise causes vibrations in the drives as well as in the air, and given how close the read/write arms are to the platters, any vibration can be deadly. There's a reason SSDs would be perfect for use in space, in aircraft, and so on. SSDs have no moving parts, thus vibration wouldn't have much effect on them. 

 

[nunim]

.. There are videos of people on YouTube yelling at big drive arrays and the noise vibrations do have an effect on disks that is quite easily visible with basic reporting tools. I don't think there was a cooling problem in the datacentre - it was just an accidental gas drop not a temperature issue.

Interesting, I never knew this, now I feel bad at all the times I've yelled at a server, I might have been causing more work for myself.

 

[rds100]

Don't fee bad, yelling is still better than kicking ;-)

 

[maounique]

They likely would. The noise causes vibrations in the drives as well as in the air, and given how close the read/write arms are to the platters, any vibration can be deadly. There's a reason SSDs would be perfect for use in space, in aircraft, and so on. SSDs have no moving parts, thus vibration wouldn't have much effect on them. 

SSDs are more prone to bit-flip errors in a radiation filled environment.

All computer parts must be insulated from radiations in space, otherwise the solar wind will damage them even in normal conditions, not to mention storms.