Man oh man....

ITNYC

Hey guys. Everything has been going great at my new job.

But...

Theres nothing like working on a extremely critical server that is time crucial and is not responding correctly and then after working on it, booting up only to see "Non System disk, Disk Error". My hairs almost jumped off my head. i forgot to properly seat the cable to the Array controller. wheew Windows 2000 never looked so good. I walk back my desk and my boss says "Is that server up!?, Dude, you alright? your sweating" lol. Its crazy when everyone is depending on you to bring that server up asap. Its like working on a bomb except it wont blow up, Just your boss, managers, Supervisors, Sys admins, users will.

Any similar stories out there?

Sie

Not quite similar but we had a panic the other day when several sites failed and panic stations were manned just to find out it was the alerting system that had failed and everything was ok!

Phew!

wd40

a very long "16 hours" story.


one year ago we had our mail server (Lotus Notes) on an old server with windows NT and a DAT drive for backup.

One day the DAT Drive failed .. a colleague connected an external DAT Drive to take the backup "at the end of the day".

For some unknown reason, the server just died ... it shutdown and "refused" to start again.

so we called the DELL people, one guy came after a few hours and asked for the the Windows NT Server CD .. we did not have that in the current location, we think that the CD is in another branch.

so we had to drive to the other Branch .. and turn the place upside down "looking for a single CD in an old IT store".

we found the CD after almost an hour, drove back and gave him the CD, he reinstalled the server "we know how to do that " .. the server started but the then the integrated NIC just would not work, even after downloading the correct drivers from Dell's website ..

we installed an external NIC -- and the drivers <Windows NT does not support PNP" . that did not work.

at that point the DELL guy left, he said the server is dead .. we might have to replace the motherboard and this will take days because we will have to order the "old" motherboard.

we cant do that .. the server must work before the beginning of the working day!.

the good news is that the server was going to be replaced any way .. we had the replacement server on site .. but we were waiting for approval for the replacement.

the bad news was that we did not have the backup for the last day and there fore we can not restore the data from the backup .. a loss of one days mail is not acceptable.

we can not use an external hard disk because we do not have one

so we had to open a PC, take it's IDE Harddisk out.. connect it to the old server and copy the data .. then connect it to the other server and copy the data.

The new server was up and running before the beginning of the working day.

The problem started @ 3:00 PM .. and it was resolved @ 7:00 AM in the next day .. 16 Hours.

KGhaleon

At the last company I worked for, I was the only one there one day when the servers went down. My boss had to go take care of something and I was the only one with technical knowledge...even though messing with the servers wasn't my job.

That was interesting...all the offices kept calling for about an hour until it was resolved. >_<

KG

sprkymrk

Lots of stories I could tell, but my favorite was one told me by an instructor I had back in NT4 days. He said his boss had been in the server room doing some scheduled maintenance on a server that needed to have it's hard drives upgraded. The boss had been gone for a couple hours and so my instructor decided to go see how things were going. He goes into the locked server room to see his boss kneeling in front of a server (thet were towers mostly) with his finger holding the power button in the "down/in" position. The boss yells at my instructor "It's about time someone heard me yelling - quick go log onto the file server, send a message to everyone that it's going down, and do a shutdown - now! I can't hold my finger here much longer!".

What had happened was that he had been at the console and did a shutdown procedure on the correct server that was scheduled for maintenance. In those days you would get the message "It is now safe to power off your computer" at which point you hit the power button to finish the procedure. Well he had pushed the power button on a very critical file server instead of the correct server by accident - but realized his mistake at the last second. As long as he did not allow the power button to come back to the "up/out" position the server would stay on. If he let it go it would do a hard shut down. So he had been sitting there for almost 2 hours holding the button in while hollering for help - but no one could hear him! He couldn't hold the power button in and still reach the console as they were several feet apart.

After that incident all the servers were clearly labeled front and back in black on yellow labels about 3 inches tall.

blargoe

ITNYC wrote:

Hey guys. Everything has been going great at my new job.

But...

Theres nothing like working on a extremely critical server that is time crucial and is not responding correctly and then after working on it, booting up only to see "Non System disk, Disk Error". My hairs almost jumped off my head. i forgot to properly seat the cable to the Array controller. wheew Windows 2000 never looked so good. I walk back my desk and my boss says "Is that server up!?, Dude, you alright? your sweating" lol. Its crazy when everyone is depending on you to bring that server up asap. Its like working on a bomb except it wont blow up, Just your boss, managers, Supervisors, Sys admins, users will.

Any similar stories out there?

Welcome to systems administration Trial by fire is the best expreience builder. I have a few stories I could share when I get some time.

JDMurray

sprkymrk wrote:

As long as he did not allow the power button to come back to the "up/out" position the server would stay on. If he let it go it would do a hard shut down. So he had been sitting there for almost 2 hours holding the button in while hollering for help - but no one could hear him! He couldn't hold the power button in and still reach the console as they were several feet apart.

Right now I'm looking at a vintage 1988 tower case with a big white power button just as you describe. One time I accidentally pressed the power button instead of the floppy eject button and was not able to reach the keyboard without releasing the button. Luckily, I had a toothpick in my mouth and jammed it between the button and the case to hold the button down. I was then able to safely shutdown the Xenix (!) system running the company's Internet email and gateway before pulling the pick.

That whole episode took only a few minutes and was not at all a panic or ordeal. But I am very impressed with anyone who has the tenacity to sit for two hours holding in a button. Fear is a great motivator!

sprkymrk

JDMurray wrote:

sprkymrk wrote:

As long as he did not allow the power button to come back to the "up/out" position the server would stay on. If he let it go it would do a hard shut down. So he had been sitting there for almost 2 hours holding the button in while hollering for help - but no one could hear him! He couldn't hold the power button in and still reach the console as they were several feet apart.

Right now I'm looking at a vintage 1988 tower case with a big white power button just as you describe. One time I accidentally pressed the power button instead of the floppy eject button and was not able to reach the keyboard without releasing the button. Luckily, I had a toothpick in my mouth and jammed it between the button and the case to hold the button down. I was then able to safely shutdown the Xenix (!) system running the company's Internet email and gateway before pulling the pick.

That whole episode took only a few minutes and was not at all a panic or ordeal. But I am very impressed with anyone who has the tenacity to sit for two hours holding in a button. Fear is a great motivator!

Maybe in addition to the high visibility labels, they could tape a toothpick to the front of each server with a sign "In case of emergency, break tape".

Kaminsky

I remember in the early days when the file server just vanished off the network and everything stopped. Ran to the server room nearly tripping over the cleaner in our rush to find out what had happened. Get to the screen and yep it's dead... Both start saying possiibilities but could hardly hear each other for the hoovering cleaner.. then it slowly dawned on both of us at the same time and we both turned to the hoover and followed the power cable back to the cabinet at the same time... YEP.. she couldn't find a socket so just quickly unplugged something for a couple of minutes so she could get the hoovering done.

From that day on, it became the job of the junior techie to hoover the now locked machine room once a week.

Looking back, it's quite funny. At the time we almost had to be physically restrained from sticking the hoover where the cleaner certainly wouldn't appreciate it.

Eventually got rid of that cleaner when she was in our office just having a dust around and washing things down. She came to my desk and got out her wet rag and started to wash the back of my dusty monitor... I instantly dived for cover. After that we banned all cleaning staff from the department except for waste bin emptying.

Cleaners can be very scary.

ITNYC

wd40 wrote:

a very long "16 hours" story.


one year ago we had our mail server (Lotus Notes) on an old server with windows NT and a DAT drive for backup.

One day the DAT Drive failed .. a colleague connected an external DAT Drive to take the backup "at the end of the day".

For some unknown reason, the server just died ... it shutdown and "refused" to start again.

so we called the DELL people, one guy came after a few hours and asked for the the Windows NT Server CD .. we did not have that in the current location, we think that the CD is in another branch.

so we had to drive to the other Branch .. and turn the place upside down "looking for a single CD in an old IT store".

we found the CD after almost an hour, drove back and gave him the CD, he reinstalled the server "we know how to do that " .. the server started but the then the integrated NIC just would not work, even after downloading the correct drivers from Dell's website ..

we installed an external NIC -- and the drivers <Windows NT does not support PNP" . that did not work.

at that point the DELL guy left, he said the server is dead .. we might have to replace the motherboard and this will take days because we will have to order the "old" motherboard.

we cant do that .. the server must work before the beginning of the working day!.

the good news is that the server was going to be replaced any way .. we had the replacement server on site .. but we were waiting for approval for the replacement.

the bad news was that we did not have the backup for the last day and there fore we can not restore the data from the backup .. a loss of one days mail is not acceptable.

we can not use an external hard disk because we do not have one

so we had to open a PC, take it's IDE Harddisk out.. connect it to the old server and copy the data .. then connect it to the other server and copy the data.

The new server was up and running before the beginning of the working day.

The problem started @ 3:00 PM .. and it was resolved @ 7:00 AM in the next day .. 16 Hours.

LOL Holy Sh*t. That must of been quite a time.

ITNYC

He goes into the locked server room to see his boss kneeling in front of a server (thet were towers mostly) with his finger holding the power button in the "down/in" position. The boss yells at my instructor "It's about time someone heard me yelling - quick go log onto the file server, send a message to everyone that it's going down, and do a shutdown - now! I can't hold my finger here much longer!".

LOL i can imagine. Thats like holding your finger in a grenade pinhole and cant let go or else Kaboom.

snadam

we actually had a scare YESTERDAY night:

Our company is hastily integrating 2 offices into one. The main hub is moving over to a satellite yard (only by name because physically this yard is HUGE). We get approval to shut down a few hours early. This gives me plenty of time to shut down the remaining servers (including critical ones like file/print, exchange, etc) and pack up all the wiring and head down to the new facility. Being proactive, the new server cabinet is already measured, shelved, and powered. All thats needed is the actual hardware. I place all the hardware in and neatly manage all the power cables (thing of beauty I tell ya), connect our KVM switch and fire up all the servers once by one to ensure that our new UPS can handle the load. Everything up to this point is working fine. HOWEVER, exchange server BIOS is whining about not being able to detect the RAID config... I calmly hit 'escape' to continue the boot process, and obviously does not detect the hard disk's. . I power down the server in hopes it comes back up okay (wishful thinking i know but Ive seen worse), and it doesnt At this point Im on the verge of freaking out. Its been a hectic 12 hour day already, and a critical unforseen error is in my hands. I rip the server out of the cabinet and start removing the case. I unplug and replug EVERY SINGLE cable that has to do with the hard disks, and even re-attached the disks back in. Everything is nice and snug, I put the lid back on, put the server in the shelf, and cross my fingers and fire it up. YAY!!! BIOS Detects the DRIVES!!!!

I wasnt really in the mood for troubleshooting a RAID array, nor hard disks, NOR restoring an exchange server. What looked like to be a long 18-20 day, turned into a 12 hour day, and I was really damn happy about it

Ive had much much worse scares than this one (Id rate it a 5/10), but its ironic this is posted and then have something similar happen to me. I guess the good news is to all of these stories is that the issues were resolved

ally_uk

Situation from previous position involving a virus

The file server went down completely and so did the Phone system My I.T manager at the time was in a bit of a rut as they were being bombarded with complaints and frustrated other managers ordering them to sort it out so after a few disgruntled words the manager ventured into the server room to deal with the issues 10 minutes pass and they return in tears.

We quickly concluded that the tears were bought on because they couldn't work the door handle mechanism to open the server cabinet ( And they were going to attempt to fix the problem??? )

As a trainee at the time all I could was sit back and watch the show and marvel at how a individual is given such a huge salary to run a I.T department but cannot comprehend how to open a door.

It just doesn't make sense

mjp81

I had a great time a few years ago with a corrupt system hive on Win2k Server. A couple of weeks previously I had been discussing the backup situation with the boss - we didn't have a proper backup system in place, and they were just not willing to invest in any new hardware. It was a pretty small company back then, and I'd only recently been put in charge of the network. Anyway, I managed to convince them to invest in a DAT drive, and they agreed to buy one at month end.

To cut a long story short, an overnight powercut restarted the server (no, they wouldn't pay for UPS either!) and I was met in the morning by a system hive error mid boot-up. Fairly quick to resolve with the Win2k disk... nope, that couldn't be found anywhere. Previous IT guy kept it with him, no-one knew where it was.

So, with about 40 workers sitting around not able to do anything, I was sent on a three hour drive to the nearest retailer that had a copy of Win2k Server in stock, then heading back to reinstall the OS.

Many many hours later, everything was fine again. But a whole day's work was wasted. Made IT budget requests much easier to get approved though!

KGhaleon

Seems like a lot of you need to start carrying copies of OS discs. ^^;
I've got a little CD-carrier that has all my operating systems. I can't imagine not ever having it. Whenever I come across a CD I don't have, I make a copy of it.

KG

dtlokee

I carry a 750GB external drive with an ISO of every CD/DVD I've ever come across. That along with common CD/DVD discs like operating systems, I haven't run into a case where I didn't have what I needed. It can be extremely frustrating when you're installing somthing and you need to add a windows component, and of course the company you're working at doesn't have a Windows 2000 disc, it comes in handy. It's also worth noting this method requires somthing like Daemon tools installed.

garv221

sprkymrk wrote:

Lots of stories I could tell, but my favorite was one told me by an instructor I had back in NT4 days. He said his boss had been in the server room doing some scheduled maintenance on a server that needed to have it's hard drives upgraded. The boss had been gone for a couple hours and so my instructor decided to go see how things were going. He goes into the locked server room to see his boss kneeling in front of a server (thet were towers mostly) with his finger holding the power button in the "down/in" position. The boss yells at my instructor "It's about time someone heard me yelling - quick go log onto the file server, send a message to everyone that it's going down, and do a shutdown - now! I can't hold my finger here much longer!".

What had happened was that he had been at the console and did a shutdown procedure on the correct server that was scheduled for maintenance. In those days you would get the message "It is now safe to power off your computer" at which point you hit the power button to finish the procedure. Well he had pushed the power button on a very critical file server instead of the correct server by accident - but realized his mistake at the last second. As long as he did not allow the power button to come back to the "up/out" position the server would stay on. If he let it go it would do a hard shut down. So he had been sitting there for almost 2 hours holding the button in while hollering for help - but no one could hear him! He couldn't hold the power button in and still reach the console as they were several feet apart.

After that incident all the servers were clearly labeled front and back in black on yellow labels about 3 inches tall.

lol classic. Like he is holding his finger on someones punctured jugular vein.

I have got too many stories to count. I don't really get nervous though.