Your biggest Ohh C**p moment?

Pash

Hi All,

So whats your biggest Ohh Crap moment youve had in IT so far?

I was on a customer site doing my rotation support, and I was testing a new offline file sync system we were working on. I was testing it using the IT managers my documents data, without making a backup (god knows why, i blame late night drinking and no throught). Yup I lost it all! I gave a call to my colleague asking for a data recovery from the last diff backup, thank god for this I still had to admit my mistake to the IT manager though, its always important to come clean with these things guys and gals!

Anyway so whats your worst ohh C**p moment in IT? Have you owned up to it or do an amazing swift recovery with nobody noticing a thing?

Cheers!

christenm123

Someone at my school asked me to look at how come their monitor would not turn on. I was thinking of all these things like the cable might be loose. So there I was behind the PC when it dawned on me that maybe the power button might be off. So then I go to the monitor and flip the switch on. I told em it was my fault and we had a good laugh.

Illusionking

This wasn't necessarily job related.

When I was 14 I had decided that I wanted to upgrade the video of my Dell 386SX25.

It had on board video that was (as I remember) very tough to disable. It consisted of removing an actual cable from the motherboard. After some work I finally got the new video card to work (it is important to point out that the case was still off of the pc). Got excited dropped the cable on the motherboard...must have hit just right, sizzle sizzle pop pop I had never seen a computer spark as much as that before.

amyamandaallen

We had a security system that had the 'users' database backed up to floppy every 2 or 3 days. Now the company that made the system didnt think to get the floppy to automatically delete the old data BEFORE it transferred the backup onto it ( so it would fit ) . Hence I had to delete the A: manually. Well lets just say that I was to busy talking to the security guard and forgot to swap c: to a:......POOOOOOF

No users information/lifts/doors etc in database so the building locked itself tight ( well 11 of them actually ) every door pad, every lift, car park gates, the lot! It was early morning so around 2000 people were on there way in!



Lets just say I NEVER did that one again!

Pash

amyamandaallen wrote:

We had a security system that had the 'users' database backed up to floppy every 2 or 3 days. Now the company that made the system didnt think to get the floppy to automatically delete the old data BEFORE it transferred the backup onto it ( so it would fit ) . Hence I had to delete the A: manually. Well lets just say that I was to busy talking to the security guard and forgot to swap c: to a:......POOOOOOF

No users information/lifts/doors etc in database so the building locked itself tight ( well 11 of them actually ) every door pad, every lift, car park gates, the lot! It was early morning so around 2000 people were on there way in!



Lets just say I NEVER did that one again!

LOL brilliant. Thats a good one. It's surprising how often IT guys and girls make mistakes, a lot of them admit it though, which is the best part. I didnt mention I was laughing on the phone to my colleague when i explained to him I had deleted everything, he was laughing and telling me not to laugh. GREAT!

keep em coming guys!

christenm123

amyamandaallen wrote:

We had a security system that had the 'users' database backed up to floppy every 2 or 3 days. Now the company that made the system didnt think to get the floppy to automatically delete the old data BEFORE it transferred the backup onto it ( so it would fit ) . Hence I had to delete the A: manually. Well lets just say that I was to busy talking to the security guard and forgot to swap c: to a:......POOOOOOF

No users information/lifts/doors etc in database so the building locked itself tight ( well 11 of them actually ) every door pad, every lift, car park gates, the lot! It was early morning so around 2000 people were on there way in!



Lets just say I NEVER did that one again!

So, how did everyone get in, or did they all just get the day off? :P

amyamandaallen

The whole point of a good IT person ALWAYS have a backup of your backups - NUFF SAID

KGhaleon

Well, back in my college days I was in this security class and my instructor wanted us to learn about DoS and various forms of attack. He often allowed us to experiment with computers that were separated from the school network. We had a single cable connecting us to the school network, so he would simply unplug it when we were working.

He forgot to once. I had all my machines with tons of command windows open, sending who knows how many pings at this one PC. I played with the ping parameters/switches and tried to put as much of a load as I could on that poor machine. We stopped that machine in it's tracks.

Later that day my instructor received an email from the administrator. <_< >_>

*whistle*

Pash

KGhaleon wrote:

Well, back in my college days I was in this security class and my instructor wanted us to learn about DoS and various forms of attack. He often allowed us to experiment with computers that were separated from the school network. We had a single cable connecting us to the school network, so he would simply unplug it when we were working.

He forgot to once. I had all my machines with tons of command windows open, sending who knows how many pings at this one PC. I played with the ping parameters/switches and tried to put as much of a load as I could on that poor machine. We stopped that machine in it's tracks.

Later that day my instructor received an email from the administrator. <_< >_>

*whistle*

LOL brilliant. Me and my colleague did the same sort of thing at a customer site unwittingly, we had their security specialist ring us up and ask if there is a DDos attack going on or something, we fell like right muppets

I love the whistle, I use that when i forget to do something or something i need to dodge out of. GREAT!

blargoe

One time I plugged in a tape drive into a Dell PERC controller and powered it up. On a server that hadn't had a full backup in about 2 weeks.

jgladwell

Not too long after I first started as a PC tech, a co worker and I were trying to figure out the net send command for some reason. We work in a domain environment with 2500+ users, and all of our "test" messages went to the entire domain. My saving grace was that I didn't use any quesitonable language and that he was involved in it, because he was my Director's "pet."

mgeorge

Well I think I can take the cake ^_^

A company that I've worked with in the past had several contractors and a guy tripped on one of those big power distrubution power cords (30 amp 125v, about an inch 1/2 thick), killed all 4 domain controllers at that company for 5 minutes, company lost an estimated 80k.

That was a big "OH S**T!!"

Personally I thought it was quite halarious

Pash

mgeorge27 wrote:

Well I think I can take the cake ^_^

A company that I've worked with in the past had several contractors and a guy tripped on one of those big power distrubution power cords (30 amp 125v, about an inch 1/2 thick), killed all 4 domain controllers at that company for 5 minutes, company lost an estimated 80k.

That was a big "OH S**T!!"

Personally I thought it was quite halarious

lol its actually quite funny how many stories there are of people doing this, some I have seen and hear of myself, it happens a lot but its still always funny

Ahriakin

Not my own but it's stuck with me for over a decade. I was working in the escalation lab for a major OEM PC retailer and they had just begun selling servers. The newly formed Server support team got to bypass normal escalation procedures and come directly to us. Had one guy walk in and say he had one of our largest corporate customers on the phone and he'd just helped install a replacement SCSI raid drive. He couldn't work out why the system would no longer boot, I asked if he'd chosen to Rebuild from the raid controller BIOS (it was an old model, needed manual rebuilding)...."Sure, we hit reinitialise and waited until it was done..."

Pash

Ahriakin wrote:

Not my own but it's stuck with me for over a decade. I was working in the escalation lab for a major OEM PC retailer and they had just begun selling servers. The newly formed Server support team got to bypass normal escalation procedures and come directly to us. Had one guy walk in and say he had one of our largest corporate customers on the phone and he'd just helped install a replacement SCSI raid drive. He couldn't work out why the system would no longer boot, I asked if he'd chosen to Rebuild from the raid controller BIOS (it was an old model, needed manual rebuilding)...."Sure, we hit reinitialise and waited until it was done..."

LOL so he deleted the RAID array entirely?

Ahriakin

Indeed

dtlokee

I worked in a datacenter with 350+ servers and a 2.1 megawatt UPS. The ups basically has a circuit breaker panelboard on it with a 400 amp main disconnect. One of the operators in the datacenter had a rolling chair and decided to wheel his way past this panelboard, in the process his knee caught the main disconnect and shut down the entire datacenter. It was bad, very bad. Why there was no cover over this setup I am not sure, but there wasn't. the company estimated the downtime from this event cost them 4 million dollars.

We later installed a cover, total cost $50. We also removed all the roling chairs.

doom969

ouch !

Was the guy fired ?

Doom969

Megadeth4168

mgeorge27 wrote:

Well I think I can take the cake ^_^

A company that I've worked with in the past had several contractors and a guy tripped on one of those big power distrubution power cords (30 amp 125v, about an inch 1/2 thick), killed all 4 domain controllers at that company for 5 minutes, company lost an estimated 80k.

That was a big "OH S**T!!"

Personally I thought it was quite halarious

Haha! Personally I think it's the fault of the person who designed the domain... I would never have all my domain controllers in the same room, if I can help it, I would not have them all in the same building....

Anyway, none of my moments seem to measure up to some of these.

I did have a user, earlier this year as a matter of fact that was complaining about their PC running really slow.... So, I took a quick look at it, and as one of the things that I typically do, I run %temp% and clear out the temp directory.... Works wonders for those people who have a gig of temp files.

Anyway.... I had no clue that this guy actually stored some of his files in the temp directory... The only things that are backed up are files they put on the NAS, so he lost everything he stored in there....

On one hand I felt horrible, on the other hand, I couldn't help but think "Why in the world would you permanently store important files in the temp directory and not have a backup!"

I tried some recovery tools but unfortunately I always have a habit of doing Shift+Delete.... None of the half-dozen or more program I tried could find the users data.

You live, you learn....

Ahriakin

Understandable mistake on the side of the user, TEMP is obviously short for TEMPoral-Stasis making it the ideal place to store important data....

DTLokee methinks you win.

Megadeth4168

Ahriakin wrote:

Understandable mistake on the side of the user, TEMP is obviously short for TEMPoral-Stasis making it the ideal place to store important data....

DTLokee methinks you win.

That's good! I'll have to remember that.

dtlokee

I wasn't there when it happened, but I had a router that was part of the frame-relay network and I forgot to save it's config. When it rebooted it was missing some of the DLCI config and we had remote sites with no connectivity. There were other issues we had to address, it's much more difficult to bring 350 servers back online than one would initially think. Order of operations was important.