What was the toughest troubleshooting problem that you solve?

s-n-c-godfather

Varies from hardware to software

White Wizard

Majority of issues I deal with I am able to solve but I did have a heck of a time figuring out a quick books issue for a client.

Quick books all of a sudden would not print PDF's and would not save as a PDF. I ran quick books diagnostic tool, uninstalled and re-installed the printer driver associated with quick books, called their support line to have them tell me that they no longer support this version and then try to sell me the latest version.

A simple uninstall of the program> reboot> re-install and the problem was resolved. I have had much more difficult issues but I have been able to pinpoint where the problem is where as this quick books issue I still am not sure what it is that caused it to all of a sudden stop printing and saving PDF's.

rsutton

I've also had some difficulty troubleshooting QuickBooks issues. We run QuickBooks in a RDS environment & store company files on a file share. We have had a hell of a time getting QuickBooks to function properly & as of today it only works because of a wonky work-around that I implemented.

Intuit Tech Support is next to useless.

jthunderbird

White Wizard wrote: »

Majority of issues I deal with I am able to solve but I did have a heck of a time figuring out a quick books issue for a client.

Quick books all of a sudden would not print PDF's and would not save as a PDF. I ran quick books diagnostic tool, uninstalled and re-installed the printer driver associated with quick books, called their support line to have them tell me that they no longer support this version and then try to sell me the latest version.

A simple uninstall of the program> reboot> re-install and the problem was resolved. I have had much more difficult issues but I have been able to pinpoint where the problem is where as this quick books issue I still am not sure what it is that caused it to all of a sudden stop printing and saving PDF's.

I think that is the norm. I have dealt with some complex issues but have been able to pinpoint and correct them usually... the toughest issue I ever had to solve took me 2 weeks and ended up being a simple hal-duplex misconfiguration on a router down the leased line (that I didnt control)... not hard at all to even pinpoint usually, but the circumstances involved and the people not wanting to admi there MAY be an issue made the problem so much worse.

ScrawnyRonnie

Not the most technical issue (or even close) but my boss and I spent hours trying to figure out why one of the laptops couldn't connect to the network using a network cable. After checking everything we could think of, I was told to just get another laptop for the user.

The next day I had the laptop sitting on my desk. Still wondering what was wrong with it I randomly thought to look at the network port on the back. Guess what I saw? The gold pins were all BENT I immediately knew why it was unable to connect...

Just curious if you all were in the same position would your mind think to check that? I know "start at layer 1", but I still overlooked that.

Chitownjedi

Massive WMI corruption through out the enterprise.

It took about a month to track it down... however it turns out there were multiple factors at play:

The Engineers before us put a bad image up on the WDS server that came off the presses with WMI corruption already (We were not told about this for a month into our investigation because the people who worked on it before felt that keeping it hidden would allow them to deflect blame, as it was now a priority issue and affected every machine in the company .. CEO machine crashed 2x during a presentation, CFO machine crashed during some presentations... they didn't want to tell the "new" guys that they put up a bad image for fear of the repercussions... but they did hint constantly that we should probably replace the image, versus troubleshooting the environmental factors."

One of those previous engineers also set up Software update groups to automatically create new software update groups in SCCM every time a new definition was created. So after a few months... there was a software update group for every endpoint protection definition, windows update, and microsoft update. We literally (By we I mean the company, i was brought in to fix it) was DDOS ing our own WMI repositories (SCCM interacts with WMI like crazy) and causing the corruption to escalate throughout the network.

I spent a ton of hours up at night research this issue and doing root cause analysis trying to figure out the problem. We kept plugging away at it and after a month we had 86% reduction in incidence of corruption. By the time we left, it was no longer a issue... the issue was going on for 6 months before I got there, and the only fix they had before that was going into CMD and typing >winmgmt /resetrepository_

It was a horrible month figuring that out. You talk about having pressure on you.. CEO over your shoulder saying Hey I heard you were our new engineer and you will fix our problems. How long will it take? A Week? Yeah.. lol.. Sure. Sigh.. lol

BillHoo

I suppose the toughest problem was one where I uttered explatives!

I was managing IT support as an Army communications officer in Iraq. My boss came to me one day and dropped a laptop on my desk. It came from one of the Military Intelligence units that is supposed to go through us for all IT issues. He said they forgot the password.

Usually this is a easy fix as any one of our techs could apply the standard admin password that we use for all the hardware we rolled out. None of our techs could get into it.

Problem. This was not one of the the laptops we rolled out. And the commo support group for that unit did not have the password either as they did not roll out that machine either. They too had trouble hacking into it. And apparently, it contained critical information that they needed ASAP and the owner was deceased.

OK. I had a tool to boot up the laptop from a CD and I could re-write the password hash.

Son of a biscuit!!

The software did not work with the laptop because there was a point in which it asked for a promt to continue - The B@St@rd had set the bios to disable the keyboard!

It wouldn't take the PS2 keyboards that we had in the shop. So I had to schlep across the base to my CHU (Containerized Hosuing Unit) and get my personal USB keyboard that I kept for emergencies.

Once I had that - Cool! The USB keyboard worked!

Son of a biscuit!

He had set the bios so that the "Y" and "N" keys were disabled! I could not continue as the prompt was "Would you like to continue- Y/N?"!

I had to go in and reset the BIOS password before resetting the bios settings that disabled the keyboard and "N" and "Y" keys.

There was one more booby trap I don't recall the specifics of, but I was finally able to reset the password and reboot it.

The unit had sent a runner to retrieve the laptop and he impatiently took it.

My boss acknowledged "I figured if anyone could get into it, it would be you!"

Thanks a lot!

TomkoTech

BillHoo wrote: »

Stuff...

Maybe easier to have a bootable usb with a linux distro on it. You can boot to that and reset the windows passwords.

Dieg0M

TomkoTech wrote: »

Maybe easier to have a bootable usb with a linux distro on it. You can boot to that and reset the windows passwords.

Officers always overcomplicate things.

rht

I had a friend come up to me with his new i7 build and told me that only 2 GB out of 4GB (2X2) is showing up on the OS. I first expected a bad module but after testing both modules on a different PC they were fine. To make matters worse i found that both modules would work in any of the four DIMs but when both module was inserted at the same time 1 module would not show up. It took me 2 days to figure out that the damn pins on the mobo's CPU LGA were slightly bent sideways causing this to happen. Although it drove me crazy at first i felt a good sense of achievement afterwords. This is why i just love PC's

darkerz

A clients car would not start, we were in a downtown area and our parking was up - they love to ticket here. This was a vendor lunch + client lunch to discuss some stuff.

I want you to visualize 4 network engineers, 2 vendors and 2 clients trying to troubleshoot a motor vehicle. Absurd theories on electrical flow, start up mechanisms, the circuitry connecting the key to the engine, grounding, piston, you name it. We even drew out the process flow for a car to start up, half of us have electrical engineering degree's. Ultimate over-shooting of the issue.

...

They ran out of gas.

colemic

Wasn't my issue, but my last trip to Afghanistan, a SCCM engineer created an isolated 'test' group to test remotely reimaging PCs... said group wasn't as isolated as he thought, and when he 'tested' his script it began the overwrite process on the systems it reached... It ran for about 15 minutes before they figured it out, and it caused between 2000-4000 systems to be affected, with a lot of generals and other high-ranking officers and SES-type individuals affected. Different sites around the country would up sending 1-2 people for about 3 weeks to get them all reimaged.

Fulcrum45

Not the most difficult issue I dealt with- but a mind numbing issue with a simple fix. I went onsite to look into a connectivity problem. The customer stated that the ISP claimed there no issues on their end and thus it was the users gateway. She said no one had messed with the device- not even to power cycle it. I did so and still nothing. I couldn't see the device no matter what and the CLI wasn't much help either. Finally in frustration I looked at the device again and noticed that the Cable Modem was plugged into the LAN interface and the WAN interface was plugged into their switch stack. *sigh* Quick swap and they were up and running. I learned that you NEVER EVER take what a customer says at face value when trouble-shooting.

Chitownjedi

rht wrote: »

I had a friend come up to me with his new i7 build and told me that only 2 GB out of 4GB (2X2) is showing up on the OS. I first expected a bad module but after testing both modules on a different PC they were fine. To make matters worse i found that both modules would work in any of the four DIMs but when both module was inserted at the same time 1 module would not show up. It took me 2 days to figure out that the damn pins on the mobo's CPU LGA were slightly bent sideways causing this to happen. Although it drove me crazy at first i felt a good sense of achievement afterwords. This is why i just love PC's

Epic

TheProf

For me, the most challenging issue I have ever encountered, would have to be a SAN performance problem which impacted our Exchange/SQL environment... most critical apps in the company!

Spent 2 days on this with support on the line and two different teams, we practically did every possible thing you can think of to try and fix the issue or even isolate the issue. At some point, the thought of building a side environment (exchange, SQL) on different storage platform was brought up.

Here's the kicker, after spending a few hours on the phone with HP, on the second day, we ended up getting another support group and they were able to locate a bad disk in the array. You might asks me, how come I did not see this myself? Reason, the array reported everything as healthy! Even the new disks that we added just before experiencing this issue were removed, thinking that some kind of RAID bug or something, and still nothing.

The worst part, these were the logs that only HP could read with their own native logging tools, we could not download it and use it ourselves... To this day, I still don't understand why they didn't have these tools available publicly.

This was years ago, today things are much better, but I'll never forget, I've never experienced pressure like I did in those two days, I've learned a lot!

White Wizard

BillHoo wrote: »

I suppose the toughest problem was one where I uttered explatives!

I was managing IT support as an Army communications officer in Iraq. My boss came to me one day and dropped a laptop on my desk. It came from one of the Military Intelligence units that is supposed to go through us for all IT issues. He said they forgot the password.

Usually this is a easy fix as any one of our techs could apply the standard admin password that we use for all the hardware we rolled out. None of our techs could get into it.

Problem. This was not one of the the laptops we rolled out. And the commo support group for that unit did not have the password either as they did not roll out that machine either. They too had trouble hacking into it. And apparently, it contained critical information that they needed ASAP and the owner was deceased.

OK. I had a tool to boot up the laptop from a CD and I could re-write the password hash.

Son of a biscuit!!

The software did not work with the laptop because there was a point in which it asked for a promt to continue - The B@St@rd had set the bios to disable the keyboard!

It wouldn't take the PS2 keyboards that we had in the shop. So I had to schlep across the base to my CHU (Containerized Hosuing Unit) and get my personal USB keyboard that I kept for emergencies.

Once I had that - Cool! The USB keyboard worked!

Son of a biscuit!

He had set the bios so that the "Y" and "N" keys were disabled! I could not continue as the prompt was "Would you like to continue- Y/N?"!

I had to go in and reset the BIOS password before resetting the bios settings that disabled the keyboard and "N" and "Y" keys.

There was one more booby trap I don't recall the specifics of, but I was finally able to reset the password and reboot it.

The unit had sent a runner to retrieve the laptop and he impatiently took it.

My boss acknowledged "I figured if anyone could get into it, it would be you!"

Thanks a lot!

LOL

Great story, curse those BIOS booby traps!

RockinRobin

BillHoo wrote: »

I suppose the toughest problem was one where I uttered explatives!

I was managing IT support as an Army communications officer in Iraq. My boss came to me one day and dropped a laptop on my desk. It came from one of the Military Intelligence units that is supposed to go through us for all IT issues. He said they forgot the password.

Usually this is a easy fix as any one of our techs could apply the standard admin password that we use for all the hardware we rolled out. None of our techs could get into it.

Problem. This was not one of the the laptops we rolled out. And the commo support group for that unit did not have the password either as they did not roll out that machine either. They too had trouble hacking into it. And apparently, it contained critical information that they needed ASAP and the owner was deceased.

OK. I had a tool to boot up the laptop from a CD and I could re-write the password hash.

Son of a biscuit!!

The software did not work with the laptop because there was a point in which it asked for a promt to continue - The B@St@rd had set the bios to disable the keyboard!

It wouldn't take the PS2 keyboards that we had in the shop. So I had to schlep across the base to my CHU (Containerized Hosuing Unit) and get my personal USB keyboard that I kept for emergencies.

Once I had that - Cool! The USB keyboard worked!

Son of a biscuit!

He had set the bios so that the "Y" and "N" keys were disabled! I could not continue as the prompt was "Would you like to continue- Y/N?"!

I had to go in and reset the BIOS password before resetting the bios settings that disabled the keyboard and "N" and "Y" keys.

There was one more booby trap I don't recall the specifics of, but I was finally able to reset the password and reboot it.

The unit had sent a runner to retrieve the laptop and he impatiently took it.

My boss acknowledged "I figured if anyone could get into it, it would be you!"

Thanks a lot!

I know I'm going to get exposed as a total noob, but I'm going to ask anyway, because I need to learn these things. What if you removed the battery to reset the BIOS to factory settings? Would that have been a way to get into the computer?

White Wizard

RockinRobin wrote: »

I know I'm going to get exposed as a total noob, but I'm going to ask anyway, because I need to learn these things. What if you removed the battery to reset the BIOS to factory settings? Would that have been a way to get into the computer?

Yes, removing the CMOS battery for a few minutes and putting it back in will reset the BIOS password.

SteveO86

I've had a few interesting things.

Had in issue with an internet connection that ended up in me debugging ARP finding out the carrier had a faulty modem that was sending GARP's with different MACs/IPs.

Had another good with a bridging loop. Turns out this network had been running a GRE tunnel in a bridge group for wired guest access going to dual 7500 controllers both with the same wired guest interfaces causing a broadcast storm every time a new PC joined that network looking for a DHCP address.

lsud00d

Diagnosing an application bug that, once realized, happened once every 45.5 (give or take a couple hours) days.

After the first crash I started recording the dates and by the third time I realized the trend. The vendors advice after engineers researched and tested the issue--upgrade to the most recent version.

markulous

RockinRobin wrote: »

I know I'm going to get exposed as a total noob, but I'm going to ask anyway, because I need to learn these things. What if you removed the battery to reset the BIOS to factory settings? Would that have been a way to get into the computer?

That should work for the BIOS password. If it's a hard drive password though, that's another story.

DoubleNNs

darkerz wrote: »

A clients car would not start, we were in a downtown area and our parking was up - they love to ticket here. This was a vendor lunch + client lunch to discuss some stuff.

I want you to visualize 4 network engineers, 2 vendors and 2 clients trying to troubleshoot a motor vehicle. Absurd theories on electrical flow, start up mechanisms, the circuitry connecting the key to the engine, grounding, piston, you name it. We even drew out the process flow for a car to start up, half of us have electrical engineering degree's. Ultimate over-shooting of the issue.

...

They ran out of gas.

I rofl'ed

mokaiba

Location: telephone call

Scenario: Lets just say, user who claims to have used windows since windows 98 and does not know what the start button is that only wants to open my companies software but it is giving an error that can only be fixed by uninstalling a program on her windows 7 machine.

Duration: four hours 30 minutes.

ande0255

This made me laugh so damn hard, well played sir

darkerz wrote: »

A clients car would not start, we were in a downtown area and our parking was up - they love to ticket here. This was a vendor lunch + client lunch to discuss some stuff.

I want you to visualize 4 network engineers, 2 vendors and 2 clients trying to troubleshoot a motor vehicle. Absurd theories on electrical flow, start up mechanisms, the circuitry connecting the key to the engine, grounding, piston, you name it. We even drew out the process flow for a car to start up, half of us have electrical engineering degree's. Ultimate over-shooting of the issue.

...

They ran out of gas.

ande0255

This made me laugh so damn hard, well played sir

darkerz wrote: »

A clients car would not start, we were in a downtown area and our parking was up - they love to ticket here. This was a vendor lunch + client lunch to discuss some stuff.

I want you to visualize 4 network engineers, 2 vendors and 2 clients trying to troubleshoot a motor vehicle. Absurd theories on electrical flow, start up mechanisms, the circuitry connecting the key to the engine, grounding, piston, you name it. We even drew out the process flow for a car to start up, half of us have electrical engineering degree's. Ultimate over-shooting of the issue.

...

They ran out of gas.

Deathmage

darkerz wrote: »

A clients car would not start, we were in a downtown area and our parking was up - they love to ticket here. This was a vendor lunch + client lunch to discuss some stuff.

I want you to visualize 4 network engineers, 2 vendors and 2 clients trying to troubleshoot a motor vehicle. Absurd theories on electrical flow, start up mechanisms, the circuitry connecting the key to the engine, grounding, piston, you name it. We even drew out the process flow for a car to start up, half of us have electrical engineering degree's. Ultimate over-shooting of the issue.

...

They ran out of gas.

I was going to write something but this post alone is epic upon itself. I can totally visualize it too!

This made me LOL out load and made my co-workers look at me odd

scaredoftests

Funny!!

bugzy3188

We run maintenance on our clients servers on a monthly basis, audits, updates and restarts. I had a particular client with Server 2008 running a virtual machine with Server 2003 to run some crappy old software that they had. This server took FOR EVER to restart, I'm talking an hour + a pop. Typically we do maintenance remotely but I had been on site and could see that the server would just sit with a black screen and no indication of what was going on before finally coming to life. I was charged with fixing this issue as the client had to do a restart themselves early one morning and were not happy. I searched for days trying to find a solution, keep in mind that this was a live environment so I really had to make a strong case each time I took the server down to test a theory. I finally had a break through when scouring the web for any sign of a possible cause and found an article about certain scenarios when running backups on a virtual machine in Server 2008 where the VSS takes a snapshot of the virtual machine, creates an object in the registry but does not properly delete it leaving orphan keys behind. Every time the server boots up it runs through these keys before finally finding the correct one. Sure enough I checked the registry where the article had told me to and found thousands of abandoned keys, I did a backup of the registry, deleted the keys and fired off a restart and BAM the server was back up in 15 minutes! I was able to create a script to automate the process that runs weekly and keeps these orphan keys from building up. I must say I felt pretty good about that one...