Does sys-admin job suck...?

JBrown

Imagine that... it's almost 10 pm Saturday night, you just walked into your apartment after a nice day of fun and BBQs and sunny beach and some good beer. Checking your emails and noticing that there are dozen of emails from Solarwinds ipmonitor stating that 1 of the production servers fallen as a hero and about 5,000 students will not be doing their finals tomorrow. The rest of the servers pretty much burning with 136F inside the rack unit. Mind you that is a brand new, Liebert MCR with an AC unit inside.
How about a security guard who has no idea about "24x7x366 emergency access" list with your name on it? Or”server room" with about 105 degrees of F in it and alarm sound that meant to say "Run Forest RUN" there is a fire in here.
It happens to be that Liebert MCR instead of cooling the rack, started pushing in hot air and the temp inside of it was around 136F/58 in Celsius. Shut down the whole thing, turned on and guess what, it started cooling again as suppose to.
According to ipmonitor logs the temperature changed from 80 to 130+ in a matter of 1 hour, I guess that is the time when AC compressor failed.
It was almost 2 AM by the time I got home, which is strange, it’s usually traffic packed in Manhattan at that time of night. City never sleeps.

NightShade03

That's funny we just went through the same thing last week....

bertieb

Sounds like a typical out of hours problem to me That, and an opportunity to review and polish up the escalation process with related documentation and staff training so they can't have any excuses for not calling out sooner next time.

TravR1

That happened to our work last summer. We lost a whole server room when the AC went out. I forget the exact temp by the time everyone got out there but it was HOT

sebster

Yeah same thing last summner, aircon died. Work hired out 3 poratble coolers to basically keep the SAN alive.

Bertieb is right though, they should of called your mobile as soon as the alert came through. Thats what our service desk does.

Piers

Pagers baby.. gotta have the pager. oh, and 24/7/366 building access authority, all keys and key cards on hand at all times.. you want our help, you hafta let us get to it, no f'n around

I've had that same heater/cooler problem several times over the years.. never that hot though.. such fun

JBrown

bertieb wrote: »

Sounds like a typical out of hours problem to me That, and an opportunity to review and polish up the escalation process with related documentation and staff training so they can't have any excuses for not calling out sooner next time.

There is no escalation process,yet, I am working on setting up one

Piers wrote: »

Pagers baby.. gotta have the pager. oh, and 24/7/366 building access authority, all keys and key cards on hand at all times.. you want our help, you hafta let us get to it, no f'n around

I've had that same heater/cooler problem several times over the years.. never that hot though.. such fun

Badgers, Badgers YouTube - Badgers oops you mean pagers, pagers. I hate those thingies. I am working to get this thingy ipMonitor :: SMS Text Pager: GSM Alert working. Hopefully it works as suppose to, next time it happens.

rsutton

Ok, I'm probably the odd one here but when I get late night emails from my monitor system stating something is down or broken I get excited. I love what I do and sometimes there is nothing else I would rather be doing.

JBrown

rsutton wrote: »

Ok, I'm probably the odd one here but when I get late night emails from my monitor system stating something is down or broken I get excited. I love what I do and sometimes there is nothing else I would rather be doing.

Well, we all do love our jobs, there is no doubt about it, otherwise we would be doing something else. It's just sucks that it happens when you least expect it to happen. It also sucks thatthis is a part of our joband its expected from us to be on call 24x7 unlike any other field.

tiersten

JBrown wrote: »

Well, we all do love our jobs, there is no doubt about it, otherwise we would be doing something else. It's just sucks that it happens when you least expect it to happen. It also sucks thatthis is a part of our joband its expected from us to be on call 24x7 unlike any other field.

You should be compensated for being on call. Also, normally you're not on call 24x7.

Claymoore

At a previous client we has a small data closet with a separate air handler that tied to the building A/C. The building A/C was terrible and we were subject to frequent outages - both planned and unplanned. To make matters worse, if the building chiller went down the separate handler would need to be restarted before it would start chilling the room again. We had an automated system that would call our cell phones in case the temperature got too high, and I used Kenny Loggins' 'Highway to the Danger Zone' from Top Gun for the distinctive ring. I never liked hearing that ringer in the middle of the night.

We were planning to move and didn't want to spend any money to permanently resolve the situation, but when the building chiller was down for 3 WEEKS (in August in Florida) we bought a separate outside unit.

Of course, we could try getting by without A/C in our datacenters
Intel: Servers Do Fine With Outside Air « Data Center Knowledge
http://www.intel.com/it/pdf/Reducing_Data_Center_Cost_with_an_Air_Economizer.pdf

itdaddy

techs and engineers in charge of a computer room like that or any 24 hour support should always have a cell phones with email/and or pager access.

There is no excuse to send 24 tech calls to a employees home email.Yikes!
That just burns me up! Where are the cell phones with internet accessss with email paid for by the company/business?

Where are the pagers issued by the company or business? This makes me crazy ..all that waste and they could have gotten to it sooner. Where I work,it is mandated to have cell phone access to employees. I am on call all the time and our AC alarms or water alarms always call our cells. I am just schocked! but yeah coming in after hours after or duing a fun party
been there done that a few times!

Big Jizay

I work in the helpdesk department at my company on the night shift, and I would NEVER email a Sys Admin and leave it at that for something like that. I would definitely call his cell phone to inform him. I would probably get hired in a situation like that if I just emailed him about it.

JBrown

We have an outsourced support for that application and they did not call or leave a voicemail to designated person about the problem.

itdaddy

nother comment I think Liebert's are way over rated big time. we have a Liebert and are old water cooler was better than our Liebert and this beast cost like 20,000.00. I think they are crap!

SRTMCSE

JBrown wrote: »

It happens to be that Liebert MCR instead of cooling the rack, started pushing in hot air and the temp inside of it was around 136F/58 in Celsius. Shut down the whole thing, turned on and guess what, it started cooling again as suppose to.
According to ipmonitor logs the temperature changed from 80 to 130+ in a matter of 1 hour, I guess that is the time when AC compressor failed.
It was almost 2 AM by the time I got home, which is strange, it’s usually traffic packed in Manhattan at that time of night. City never sleeps.

where in manhattan? i work nightshift for a financial firm over at 50th&3rd and the streets are dead around 11PM around this way....

SRTMCSE

itdaddy wrote: »

nother comment I think Liebert's are way over rated big time. we have a Liebert and are old water cooler was better than our Liebert and this beast cost like 20,000.00. I think they are crap!

+1 Liebert blows, their UPS blow and their crappy Multilink software blows the worse. for an "enterprise" application i've spent the last 3 days manually configuring hundreds of servers since management decided to change our emergency shutdown ordering. i might blow a gasket if/when they decide to upgrade from v3.7 to v4.0

binarysoul

I don't think the job sucks unless the person wants it to suck.