Biggest screw ups you've made?

KGhaleon

Last year I was working on a desktop PC for a customer and they called me that morning asking how the work was going. I informed them that I was done installing the operating system. I was going to install the drivers, software, etc next.

They took that to mean that it was ready to be picked up, and promptly drove down to our office while I was on my break. o_o

I still had a lot of other work to do on the system and told him I wouldn't be done for another hour or two. He lived 30 minutes away, so he decided to sit down and wait until the work was done, despite needed to finish work at home for his job.

I've learned to word these things better when speaking to people. >_<

KG

cdad2000

That's right, especially with non techie people. I worked desktop support for the 911 system, and one busy day I accidently powered down a PC while she was in the middle of a call. I'm a dope!

Schluep

I have auto spell-check enabled on my outbound e-mail through Windows Mail (formerly Outlook Express). When I hit send it prompts me regarding each spelling error and then immediately sends the e-mail once each one has been addressed. I was sending an e-mail to a client regarding an issue we were having with one of their employees who we communicated with on a regular basis.

There was only a single spelling mistake flagged as an error in a message I was typing and it was the name of the Company to which I was sending the e-mail that was not recognized. I accidently hit "Change" instead of "Ignore" and it changed their name from ______ Services, Inc. to Communist Services, Inc. which was very close to their actual company name. The e-mail was going out the door and I was trying to stop it. I tried to move it from my Outbox to my Drafts folder, but it went out to fast and I missed it. I didn't want to offend them in thinking it was intentional so I quickly followed up with a play mentioning my mistake and trying to make light of the situation. Since my message was worded rather strongly and it was a major issue we were facing I definitely did not want it to be mis-construed as an intentional attack on them.

A week went by without a response.

I called to see if my message was recieved since I had it marked for a reciept that did not come back and found out that the person I sent the message to had resigned three weeks earlier and the messages were lost when they disabled his e-mail account. A co-worker of mine whom I had copied on the e-mail really enjoyed it.

Netstudent

I installed a voice script on the wrong IVR system once because the IVR machines were on a KVM switch. I was using my keyboard to bounce around the desktops. All the desktops for each IVR were identical. Well one of the other IVR machines already had a script CD in the cdrom from weeks past. When I jumped to the desktop and saw a script CD in the cdrom, i automatically assumed it was the one I had just popped in the cdrom. IN actuality I was looking at the wrong desktop. SO I was installing the script over and over because it wasn't the new one I had just ordered. I was operating the wrong IVR machine. Then I called our vendor and was like "hey you gave me an old CD". Got them to send out a new one, then finally realized I was installing the old cd on the wrong IVR. Dumb ass!

Lesson learned: Have a sheet with your KVM port numbers and connected desktops if you daisy chain your KVM's.

Lee H

Hi

Not the biggest mistake one could make but more on the bad luck side

A teacher in one of my old jobs brought her PC in, small form Compaq, she said it takes about 5 inutes to come on

I turned it on and nearly fell off my chair when the system tray filled up with about 30 icons

I recommended that i re-install windows for her at a cost of £25, while her PC was in my possesion the hard disk started to constantly make a clanging noise

A new hard disk for the PC cost £40, so i was at a loss of £15.


Lee H

dynamik

Netstudent wrote:

Lesson learned: Have a sheet with your KVM port numbers and connected desktops if you daisy chain your KVM's.

This is a handy utility for keeping track of what windows machine you're on (you can configure it to show a bunch of other useful info as well): http://www.microsoft.com/technet/sysinternals/utilities/bginfo.mspx

I have it on all my machines at work as well as the VMs that I play around with.

liven

I got a couple

Buddy of mine wrote a script to restart a daemon if it died. Well it was a C script that forked the process.

Not fully understanding the script (he usually writes shell scripts) I put the script in cron to run every minute. This is the way we usually did it at that company. The script e-mailed on every restart.

Luckily I was the only e-mail recipient becuase I started getting FLOODED with thousands of e-mails. Needless to say I realized my mistake and turned it off.



This other instance was not my fault but it was pretty funny.


About 7 years ago when I still worked on the help desk I worked for a small ISP. We did all kinds of networking and web hosting. I worked weekends in order to go to school during the week. I was the only weekend tech/noc/support/admin etc... Well one day I was on the phone with someone when I turned around to see a police officer standing behind me. He waited till I got off the phone and then asked if I had called 911. I insisted that I had not, and he was pretty cool about it. He stated that that dispatch was getting hundreds of 911 calls from our building. He even radioed dispatch over his walkie talkie ( I was able to over hear the dispatcher) to confirm if the calls were still coming it. The dispatch lady was P!SS3D!!!! You could here her yelling stating that she was just going to ignore the never ending calls. The office looked around just to make sure nothing was wrong and left.

Turns out that one of the dedicated web hosting customers had a modem in their server. They also had some sort of automated monitoring software running that would place an outbound call upon a certain situation. Well who ever set this up, used a dummy number (they though they had just typed random digits) that started with 911... And then the triggering condition was happening endlessly in a loop. This caused the server to call 911 over and over, but there was nobody for the 911 dispatcher to talk to.

I don't know if our company got fined and if we did i am sure it was passed on to the bonehead customer.

All in all it was pretty funny...

bertieb

Several years ago, after moving from an entry level network support role and being promoted to the new kid on the block for the Microsoft support team, I decided to show how keen I was to my new workmates. I decided to install an updated monitoring tool/agent to both nodes on a MS Win2000 SQL cluster system that had been on the cards for weeks. In the sillyness of my youth I decided to just go ahead and do it without informing the customer or scheduling planned maintenance. Well, I'd tested it on the dev system, it didnt need a reboot and ran just fine, so what could go wrong?..............

Well, It worked just fine for the inactive node so I decided to install it on the live node without failing the cluster services over. One BSOD later with the cluster services 'hung' in limbo land, forcing me to run over to the datacentre across the road to shut the broken cluster down, reboot each node in safe mode and disable the monitoring tool service before booting everything back up and resuming normal operation. All said and done, it resulted in a 15 minute period of total loss of business for the client. Somehow, they didn't notice and I was walking on egg shells for the next 6 months with the thought it might pop-up at any moment. If I knew what I know now, I would have told them and suffered the consequences - believe me, looking over your shoulder every day for a few months expecting to see an irate blue chip client waving an axe (or the boss with a P45 form) doesn't do your health any good. I did get a rather stern talking to from everyone else in the team, and rightly so!

Still, always look on the bright side, so I now consider that episode to have been 'excellent character building stuff'

Lesson learned - Change Management/ITIL best practises rocks

Daniel333

My biggest mistake? "Reply to all"


Biggest mistake I was a part of, was a mistranslation from Chinese to english by our front end guy. Work order said "No backup needed, format hard drive, install SP2, apply security updates and install office 2003 and activate it" After the lawsuit, turns out all he wanted was "install office". Geek Squad was fun like that.

Pash

Delted one of our customers IT managers Shared drive. Thank god for backups is all I say to this

sprkymrk

I've made so many, how about a Top 10 List? Some of these are from my high school days or my days as an electrician before getting into computers:

10. Skipping school with a friend whose dad was a mean lawyer.
9. Not going to college when dad would have paid for it.
8. Complaining about having to work OT when the boss was within earshot.
7. Replacing a 75KVA 480/277V transformer to fix a problem - and not have it fix the problem.
6. Trying to save time at the expense of safety and falling from a height of 14 feet onto pavement, fortunately my head broke my fall.
5. Driving too fast on a country road at night around a curve while it was snowing.
4. Similar to Schluep, I accidentally let spellcheck - incorrectly - change the spelling of an important customer's name; "Dear Mr. Honeybear"...
3. Not double checking a co-worker who was assisting me in setting up a new site. I did the AD and firewall stuff, he did the Exchange 2K server - and while trying to trouble shoot a problem he set it as an open relay.... A month later I had to fly back to the site to correct it and then spent the next two weeks trying to get them off every ISP's blacklist.
2. Going behind the VP's back to go to DefCon in Vegas one year.
1. Not going to tell....

JDMurray

liven wrote:

Well who ever set this up, used a dummy number (they though they had just typed random digits) that started with 911...

This happened in a place that I worked, where all of the internal telephone extensions started at 9000 and went up as more lines were added. It was eventually discovered that calling extensions 9110 through 9119 was interpreted by the local PBX as dialing the emergency 911 number. You would think those extensions would be disallowed by the PBX firmware by default, but they weren't.

snadam

actually believing that our ISP would complete the PPTP WAN link and DIA setup on time, without any intervention from my part...twice...

JDMurray

dynamik wrote:

This is a handy utility for keeping track of what windows machine you're on (you can configure it to show a bunch of other useful info as well): http://www.microsoft.com/technet/sysinternals/utilities/bginfo.mspx

Whoa...sweet! I always handmade wallpaper bitmaps like this using MSPaint specifically for IDing remotely-managed machines. I wish that I'd thought of writing a program like BgInfo ten years ago.

Slowhand

I think I'll follow sprkymrk's lead, and do a top-ten list:

10. Putting a screw through the chassis of a laptop when reassembling. (In all fairness, the screw looked like the ones that came with the system, but it had been placed near where I was working and I didn't check it closely enough.)

9. Calling and telling a customer that their computer was ready to be picked up when, in fact, nothing had been done on it aside from corrupting the OS install and losing all their data. (My mistake was calling, not breaking the thing.)

8. Running at least ten customer hard drives through a hard drive tester before realizing that the device was frying the drives. (Man, we sold a lot of hard drives that month.)

7. Blowing away RRAS on a server that was using it for VPN traffic, instead of re-configuring the interface RRAS was using.

6. Getting so mad at a printer that wouldn't stop jamming that I "accidentally" snapped the paper tray in half.

5. Replying to my girlfriend's dirty email and accidentally CC'ing the engineering department. (Who never let me live it down. They still call me 'Love Machine' to this day.)

4. Rebooting a machine that was in the process of serving up programming to a television station, instead of rebooting the primary server that had frozen up.

3. Downloaded a torrent of FreeBSD 6.2, eating up the entire OC48 connection for the whole company for about ten minutes. (I needed the image to install a lab machine, but I forgot to change my duplex settings, as not to overload the buggy switch. Got a GREAT download speed, though.)

2. Leaving a "permit any any" rule on a firewall after troubleshooting a communications issue, as the first entry, for about 24 hours.

1. And the number one screw-up: partaking in downing internet connectivity for not only our 500 managed and co-located clients, but the greater part of the county, as well. Worked for a Tier-1 ISP/Datacenter, the Sr. Engineer was working on putting down a DOS attack, and we were supposed to change the ACLs on a couple of VLANs, as well as on the incoming ports on two routers. Between the four of us working on it, we mixed up the ACLs, ended up blocking access between the routers and switches, and took out all traffic on all paths. The result: the Sr. Engineer scrambled out to the switching room with a laptop and a console cable in tow, and we discovered that the VoIP system was not on the same internet connection as the servers and our routers.

snadam

JDMurray wrote:

dynamik wrote:

This is a handy utility for keeping track of what windows machine you're on (you can configure it to show a bunch of other useful info as well): http://www.microsoft.com/technet/sysinternals/utilities/bginfo.mspx

Whoa...sweet! I always handmade wallpaper bitmaps like this using MSPaint specifically for IDing remotely-managed machines. I wish that I'd thought of writing a program like BgInfo ten years ago.

BGInfo is your friend we just implemented this a few months back and its really handy!

Slowhand

Oh, I did forget one thing on my list. At my last job, while I was doing some late shifts at the NOC, the CEO walked in on me one night with my feet up on my desk, smoking a Gispert Churchill cigar, playing a video game. He looked me square in the eye, scowled, and said "You'd better have one of those for me", then laughed.

After my heart re-started, I gave him one of my cigars and walked outside to chat. The only "trouble" I got into was that he told me not to smoke inside the building again. Or, if I was going to, not to let the VP catch me doing it. Just goes to show, there's a fine line between sharing a smoke with your boss and updating your resume.

jarjar

Slowhand wrote:

Just goes to show, there's a fine line between sharing a smoke with your boss and updating your resume.

-Oh, that's good. You just hit the proverbial nail on the secret universal truth head.

Schluep

Most of my worst IT mistakes were due to me rushing with e-mail (like the one I posted above) despite working with all types of Servers, Databases, and critical GIS files. This one actually had the potential to be worse than the one listed above (though not as funny).

Another bad one was when I was sending very detailed reports created from our database for two large clients to review. Contained within that data included the addresses of all of their customers. They were two big competitors. I accidently mixed the links up on my e-mails and provided them with detailed download instructions to get onto our highly secured servers containing their competitor's critical data.

After about 20 minutes I got a bad feeling about it and went back to read over both e-mails and discovered my error. I was able to remove both links (quicker than fixing them due to security measues in place) and create new ones just before I got the reciept back on their e-mails. I apologize for sending them "broken" links as a part of the instructions and gave them the correct ones after verifying again that neither had accessed the servers.

This would likely have resulted in a huge lawsuit. Thankfully God was looking out for me that day.

RussS

I refuse to comment on the grounds that I will inextricably incriminate myself

ally_uk

Running a customers machine and forgetting to fix on the Cpu fan lol resulted in a fired CPU

The otherday I was fixing a pc and placed in my workbench didn't realise a screw was underneath the unit powered on the beast and bang ended up with a shortened board,

I hosed countless amounts of supplies mainly E machines ( Cheap nasty PSU's )

Accidentally changing the boot order of a machine to boot from A: and when the error came up unable to find boot device I spent a good ten minutes staring in puzzlement ( O/S was on C: drive )

Hosing many linux boxes by accidentally removing elements of the kernel through aptitude! lol

deleting recovery partitions accidentally in windows OEM machines I was in a rush wasn't paying attention.

Countless amounts of networking mistakes i.e cables not plugged in etc, stood there like a lemon wandering why I can't connect to anything.

And the classic whilst being on helpdesk was " Ally my machine wont turn on " you run down 4 flights of stairs to realise they havent plugged it in or they havent flipped the power switch on the back of the supply lol

pryde7

Waoh!!! You really thought about this screw-up issue.
Picture this scenario...
I wanted to use my tech expertise to make the first impression to a secretary I've been eying!
I was sent to their office (client site) to install some graphic programs. I played around to wind time so that I can finish only when she will be going out for break!
You know what? Before then I knew nothing about deepfreeze. I spent 3hrs...after each restart I went to the programs menu and could not find the new programs....hmmm!!!
She could not go out and I could not leave without the programs working
My heart was running in safe mode...that was the day I was most embarrassed.
Finally I used some dll words and corrupt registry stuff to convince the manager and took the pc to our lab. Only then I discovered it had deepfreezed installed. Though I returned the pc as a hero only me knew how pissed I was.

garv221

My company has about 120 vlans including corporate, public access, library,random hot spots and some other departments. Well, some corporate offices in a building were being turned into study areas for college students. I switched the vlans to their respected access and went on with my day. Was checking network intrusions and reading logs when I discovered DHCP was issuing ips to random pcs outside of our local domain. Yeah, I mixed the vlans up and set public college kids up with corporate network access. None of the kids knew it and it was only for a day but still I spent soooooooooooooooo much time and money building a secure network only to have it blown wide open in less than 30 seconds by its creator with a simple "switchport access" command. This is why I have implemented the "no huge changes on Friday" rule.

Schluep

garv221 wrote:

This is why I have implemented the "no huge changes on Friday" rule.

Have been using the "No huge changes on Friday/Monday, or after 1:00 p.m." for the past few years now. I avoid Friday for the reason you mentioned and Monday because it is our busiest day (don't want to break something then). After 1:00 p.m. is because mistakes seem to take a few hours to be discovered and usually show up after I am gone for the day (and I don't want to have to go back).

binarysoul

JDMurray wrote:

liven wrote:

Well who ever set this up, used a dummy number (they though they had just typed random digits) that started with 911...

This happened in a place that I worked, where all of the internal telephone extensions started at 9000 and went up as more lines were added. It was eventually discovered that calling extensions 9110 through 9119 was interpreted by the local PBX as dialing the emergency 911 number. You would think those extensions would be disallowed by the PBX firmware by default, but they weren't.

Did any call actually went to 911? I think there are penalties for calling 911 even by mistake.

win2k8

I biggest screw up that ever happend to me was while working my new first job as a tech for a small company's headquarters, I basically messed up their whole production server, which at the time was only 1. It had 3 SCSI drives, 1 with the OS, and basically it died. And this company had no backup solution, but thank gawd atleast their data was on a seperate drive

But anyways it took me an hour just to figure out which drive went into which slot which was associated to the right controller for the harddrive. Only after trying all possible combinations realizing the whole server was dead, and well felt kinda guilty for killing it, even though i had done nothing to kill it. The boss came in and i was like em, the server just died.

It turned out alright though, we ordered another server with SATA drives, thank gawd, i hate SCSI chains... had to create the user accounts and groups and shares all over again though, with having no backup, and transferred the data back into the server, renamed it to the old production server, and everything was back and ruuning. Plus the boss realized this near death experience of the company, and got a cool external network area storage drive.

So while i'm glad its over with and hopefully i'll never have to experience a producation server die again, I learned a lot from it, from how to stay calm, and most importanlty to always have backups, when one day ur harddrive fails on u or the whole server itself.

win2k4

Schluep

binarysoul wrote:

Did any call actually went to 911? I think there are penalties for calling 911 even by mistake.

I don't know if it depends on the State or if it the same nationally. We have a large Call Center in the building here and when dialing 9-1 to get an outside line people frequently press the second one twice by accident. Rather than wait on the line and inform them it was a mistake they hang up very quickly when they realize what happened. The local police department here is required to investigate any such calls on-site since a potential attacker could have hung up the phone and then answer the return call to state there is no problem. Two police officers show up at our front door every time it happens and I believe we are charged for it.

mobri09

Failed over Unity by accident....Good thing it failed over properly

JDMurray

binarysoul wrote:

Did any call actually went to 911? I think there are penalties for calling 911 even by mistake.

Oh yes, that's how the company found out those extensions needed to be blocked at the PBX. The 911 system is controlled state laws, so the penalties for misuse will vary from state-to-state. However, ours was an accidental misuse--I'm sure not the first time its happened to a company using that brand of PBX--and there was no penality.

manny355

dynamik Wrote:

"This is a handy utility for keeping track of what windows machine you're on (you can configure it to show a bunch of other useful info as well): http://www.microsoft.com/technet/sysinternals/utilities/bginfo.mspx

I have it on all my machines at work as well as the VMs that I play around with."


We utilize this as well at my work and for the most part it works great.

Other than fudging up time of completion of projects I haven't had too many issues...(i'm knocking on wood now...lol)

liven

binarysoul wrote:

JDMurray wrote:

liven wrote:

Well who ever set this up, used a dummy number (they though they had just typed random digits) that started with 911...

This happened in a place that I worked, where all of the internal telephone extensions started at 9000 and went up as more lines were added. It was eventually discovered that calling extensions 9110 through 9119 was interpreted by the local PBX as dialing the emergency 911 number. You would think those extensions would be disallowed by the PBX firmware by default, but they weren't.

Did any call actually went to 911? I think there are penalties for calling 911 even by mistake.

Yes hundreds of them went to 911.

I think it was one per minutes for hours.

The police officer explained that the dispatcher was getting flooded with calls. He then called her on his walkie talkie thing and I could here how P.O. she was.

As soon as the Police man left I called the head of the web hosting department. He was aware of the customer. He knew which one it was because we have to install a pots line to the data center for this particular customer. He called them and had them fix their server ASAP.

I never got the details as to if there were any fines etc.

It was really weird when it happened because I thought I was in trouble. However, it had nothing to do with me. I didn't get any more details after that.