Definition of a bad day...:-(

it_consultant

I was doing work at a remote site (connected by single mode fiber) and the ports on the patch panel went dead. This was predicating a move from an omnitron --> HP Procurve for fiber termination to patch panel --> directly into Brocade optic for fiber optic termination. That part didn't even get off the ground!

It is a sad day in your adult life when you are gently squeezing a backbone fiber optic cable so someone 40KM away who has an OTDR device can measure signal loss. Of course, we told everyone "yeah, we will only be working for about 4 hours...".

Sometimes it goes so well, and sometimes it goes so badly.

Claymoore

Had one of those weeks recently. I spent a combined 66 hours waiting while trying to copy a single VM for a test deployment in Azure. Click a button, wait 6 hours, need a change, wait 6 hours, copy a different setup, wait 24 hours...

Roguetadhg

Why were they bad days?
- Besides the possibility of breaking glass
- Time Spent to copy.

Were you making changes on the international space station, Claymoore? That's a heck of a long time to wait for feed back from a click.

it_consultant

I should have added that the remote site went down and their was was no backup link. I just learned that their isn't another full fiber run from that location to the central office, we will have to jumper it through another remote location. What a PITA.

Roguetadhg

Ah gotcha, now it's clear! Did you squeeze too hard and have to splice the cable during production hours?

N2IT

I am managing a service desk in an environment with no service portfolio or mature change management.

Always a good time

paulgswanson

I am managing a service desk in an environment with no service portfolio or mature change management.

Ouch thats harsh it took us years to clean that mess up on our side.

RouteMyPacket

You think that's bad? You don't want to hear about my day

Roguetadhg

We do.

I was studying Linux+ while hearing drama about co-workers that don't show up for work :P

Also found 2 new wallpapers for my Linux Distros. Awesome.

Claymoore

Roguetadhg wrote: »

Were you making changes on the international space station, Claymoore? That's a heck of a long time to wait for feed back from a click.

I was the uploading of the VHDs to Azure that was killing me. Upload an 11GB VHDX, learn that's not supported, upload VHD, find that I uploaded it as a block blob instead of a page blob, upload again, find that only fixed disks are supported, convert and upload a 40 GB file. And I wasn't getting anywhere near the 5Mbps upload I am paying for. I have a list of the steps I went through that I plan on posting on my blog. It's funny in a glad-that's-not-me kind of way.

it_consultant

Oh yeah, I don't do anything in VHDX yet. I have already deployed Server 2012 Hyper-V too.

it_consultant

Roguetadhg wrote: »

Ah gotcha, now it's clear! Did you squeeze too hard and have to splice the cable during production hours?

No, the backbone was fine, the jumpers were toast...which is easy enough to fix but it took a while to determine that was the culprit. I had the pit in my stomach fear that it was the backbone.

jibbajabba

Worst day I had was a server failure at 3am (always seems to be at 3). 90 minute drive to the Datacenter swap power supply. Noticing a raid failure. New disk doesn't rebuild. Bust stripe so array needs rebuilding, which requires imaging. Only solution available (Win2000 server), cold Acronis. Ever imaged 250GB over 100Mbit and onboard raid with screwed array? Thrown a Cat5 to a gig switch. After hrs. on the floor, Acronis crashes with unknown error - of course at around 85%. Trying to P2V instead. After a few more hrs finally a working server. Needed to reconfigure most of the network as physical server was on an old vLan which wasnt presented on the esx infrastructure so had to reconfigute all switches and core, clear arp all around and whatnot. Got the initial call at 3am. Was home at 9pm. The next morning I was told that the accounts department forgot to tell us that the customer cancelled the management weeks ago (which meant he had no warranty any more) and also cancelled the service for the following 1st of the month and already moved his website he hosted on that server to another provider. Fun fun fun

f0rgiv3n

jibbajabba wrote: »

The next morning I was told that the accounts department forgot to tell us that the customer cancelled the management weeks ago (which meant he had no warranty any more) and also cancelled the service for the following 1st of the month and already moved his website he hosted on that server to another provider. Fun fun fun

THAT is HILARIOUS!!!!

rsutton

I had a bad day a week ago:
- Went to data center to replace failed drive in a RAID5
- After inserting the new drive and rebooting there was a power spike
- Rebooted a second time and found that I had lost another drive
- Thought I would just replace that drive too and restore from backups
- Backups had been failing for 2 months =[

it_consultant

rsutton wrote: »

I had a bad day a week ago:
- Went to data center to replace failed drive in a RAID5
- After inserting the new drive and rebooting there was a power spike
- Rebooted a second time and found that I had lost another drive
- Thought I would just replace that drive too and restore from backups
- Backups had been failing for 2 months =[

So, in the middle of our fiber crisis, we had a server experience a triple drive failure...no regular backups. Luckily someone had copied the config file down anticipating reloading the software. For the weekend our road signs showed a warning for roadwork which had already been completed.

DevilWAH

Never have bad days, there's the tedious days when you just don't feel like you have achieved anything as you drive home. And the exciting days where so workman cuts through a bunch of fibers, or the fire system kicks of kills the power and fill the server room full of foam..

There was the day we had to upgrade about 300 Cisco 6500's in a day due to a software bug, on a live network.. team of about 12 of us trying to get through them all with out causing a single network outage. A lot of repetitive task, with no space for a typo, in a tight time frame, knowing that any outage cost the client £millions per minute. Yer that was fun

Things always going to go wrong some times, either through some ones stupidity or just case some times they do, as long as its not my fault, enjoy the madness unfold.. And if your lucky your get to learn some thing new.