The result of a bad draft and a firmware update
2:13am Phone rings with a very concerned person on the other end of the line. “The server room is full of smoke, we think a server is on fire”
In a groggy state, I ask the obvious question: “Is it an emergency? do you want me to come in and check it out?” response: “YES!”
Considering myself to be a daily firefighter- I didn’t think much of the situation, so I drove on the deserted streets to work. Upon quick glance, I did not see any flames from outside the building- which I took to be a good sign.
I get escorted up to the server room by a panicked guard. I ask the reasonable questions of “how is your night going?” to lighten the mood.
I should take a moment to explain… most offices keep their server rooms where the business folks do not have to see or smell the IT folk, but this was a new concept building that believed in highlighting the technology advances in a prominent fashion.
Upon walking into the server room, I notice the smoke monitoring system is indeed alerting on a higher than average smoke concentration. Still no flames visible.
Going off of the basics, I am smelling for a pleasant “magic smoke” essence as I walk on the hot aisle. Much to my dismay, I am unable to locate the scent I was looking for. Instead I am able to get close enough to the air intake to the room to smell it is coming form the roof HVAC units.
A splendid opportunity as I had yet to test my access to the roof. I begin crawling into the access doors of the HVAC units- trying to trace the smell, and eventually follow them to the inlets from the outside.
Looking around- I spot the source: the generators were spewing diesel soot into the air- all part of the regular maintenance.
This morning was a special morning- the wind had been blowing from the West which directed all of the exhaust into the server room. As it turns out- the building management had run a firmware upgrade which reset the generator testing schedule to a time that closely matched 2am.