Definition of a bad day...

This is mostly just a rant because I've had a HORRIBLE day today and I wish I was nursing a beer while writing this but I'm sadly still working. I just want to preface this by saying we all have bad days and I love my job so I wouldn't trade my job because of a day like this. I'm definitely changing or switching around any potentially identifiable information in this post to protect the innocent

Today I was working on a new site build-out that I've been working on for the last month. I took it over from a senior network engineer who had quit mid-project. When this thing was handed to me, the engineer had ordered wiring for the office, MPLS, and redundant DSL and procured a $25K quote for network equipment for this tiny little temporary office that would be shutting down in 6 months. He hadn't spoken to the end-users or management about the needs and it was a bit of a cluster when it got to me. Very important: The only things I knew about this site were provided to me by him and his notes.

I spoke to the managers at the time, reassessed their wiring, equipment, and data needs and ended up with a quote for about $10K worth of equipment and changed the MPLS line to dual T1 lines since the install time would have been a few months longer for the MPLS to be installed (bureaucratic service provider and budget issues). I called up the service provider at the time and asked him to make the adjustments on the orders for DSL and the T1s but I did not change ANY other information provided by the Sr Engineer because he knew more about the site than I did. I ordered a nice little 2921 router with a PoE switchport module and some VoIP phones. When I reviewed the quote, I noted that the PoE power supply was not on it and specifically e-mailed the vendor to ask about it. He put my worries to rest by insisting it was included in the price.

Fast forward to a week ago - I get an e-mail from the service provider saying the T1 circuit and DSL is installed and ready to be turned on. While I have the VoIP phones and all the modules for the router, I do not have the router. I put pressure on the hardware vendor and he FINALLY finds out what was causing the delay and rushed it to be delivered to me. I received it on Tuesday with the growing pressure of knowing that the employees are going to be moving into this site on 11/15. I needed to hustle to configure this router for voice, routing and switching but I got hit with a major outage yesterday that kept all of us engineers busy until late at night.

Cue to this morning - First thing this morning, I get a call from our construction team manager who is surprised that the service provder didn't let him know the circuit was installed even though he was the contact. The reason being is that while he did run a line from the MPOE to the office, he did not connect it to anything because the SP was supposed to call him. It's going to take some time for him to get his tech out there to connect it and I'm running late configuring the router. While I'm working on the router, I can't get the switchport module to power ANYTHING over ethernet. About four engineers and I toiled on it for a good 3 hours only to realize that the vendor didn't send it with a PoE power supply. I end up having to run down 5 flights of stairs to our storage room, procure a switch from another project to use for this one temporarily, and configure it on the fly. After that, I FINALLY get everything in my car and run over to the site since it was close enough to our corporate office.

I'm onsite and connecting everything but we still don't have the tech onsite. After about 45 minutes, he finally shows up and goes to the MPOE to extend the line to the T1. After about 20 minutes, I get a call saying they cant find the circuit. I give him the circuit information and went back to my work. 30 minutes later, I get another call saying that they still can't find the circuit. After spending the next 5 hours going back and forth with the LEC, SP and onsite contact, we find out the problem: The original senior network engineer submitted both the DSL and T1 orders with VASTLY different addresses than the office's building's address (it's located in something else so it's very easy to do and very complicated)

So where we at tonight? ETA to fix this is 30-60 days. I had to explain to the CTO and IT director what exactly happened when I got to the office at 7pm and we're going to have to announce to about 12 employees that they cannot move into their new office for a couple more months. THIS is a bad day.

So... I'm not in any trouble and no one is blaming me but it was a cluster today. I still love my job and what I do. This was just the perfect storm of crap today.

/end rant

Find more posts tagged with

Save $250 on 2025 certification boot camps from Infosec!

Book now with code EOY2025

Button

Comments

DPG

I've definitely had plenty of these type of days working in IT. They usually make you want to quit and never have to deal with another electronic device again. At least there are usually way more positive days in which you learn something cool or get to save the company's ass.

emerald_octane

It looks like you handled it with class. This may end up looking better for you in the long term because you deal with these massive issues as they come. I wonder if the old engineer QUIT or was he FIRED. Sounds pretty incompetent.

If it's any consolation, i've been in hot water with my CEO over why I couldn't convince the phone company and comcast to move our T1 and CATV lines....over a HOLIDAY weekend (4th of july, which I missed because he wanted no downtime). Don't forget we are members of (ISC)2. In theory, we can handle anything LOL.

demonfurbie

welcome to network design and dealing with subs

the easy part is config/design the hard part is dealing with the "others" (contractors, users, service providers, ect ect ect)

i have gotten to the point where i ask for blueprints of the building before i start any kinda planning and i have the all the equipment delivered to me (not wiring ... duhhh) and i go plug it in my self.

and when dealing with service providers and contractors make sure you get a meeting with them first via phone or in person so they can all be on the same page with what you have planned, this also helps iron out any issues that both may have

Iristheangel

We're an international organization so plugging it in for myself in every site doesn't work. Usually I have the equipment shipped to me and sent onsite but this was a special case that happened to be so close and we were approaching the deadline, I went onsite for this one time. Anyways, it was a cluster... I didn't start this project, the original specs were not correct and there were all sorts of delays before I could actually order the equipment (budget squabbles - the original $25K quote ruffled a lot of feathers)

Kaii

truly bad day but still love your job

.

happy to hear it ^_^

m3zilla

DPG wrote: »

I've definitely had plenty of these type of days working in IT. They usually make you want to quit and never have to deal with another electronic device again.

The worse part is when you realized what happened, and you wonder how you could have missed the issue, or screwed up so badly!

NotHackingYou

Sounds like a bad one!

truckfit

you survived

what doesn't kill you make you stronger.

Nice one

D-boy

I guess my day was not so bad after all

uyen_nguyen

A bad day is when teachers give unexpected pop quiz.

boredgamelad

It wouldn't be a pop quiz if you were expecting it, now would it?

Iristheangel wrote: »

So... I'm not in any trouble and no one is blaming me but it was a cluster today. I still love my job and what I do. This was just the perfect storm of crap today.

Thanks for sharing the story. As crappy as that day was, it sounds like you're doing exactly the kind of work I'd like to be doing someday.

stlsmoore

Yea rough days can be stressful.

Today for me lost another module on our core nexus switch again for the second time within two weeks. Just found out the rack that this device is in is currently oversubscribed on power, it's currently at 110%! Think I found why this issue keeps happening at least, on hold now with TAC to verify.

Roguetadhg

Double Facepalm.

My bad days revolve around: "Tomorrow we have a client coming in. We need to have 40 booths setup, imaged, and ready to go."

RomBUS

Wow sucks that you were left holding the bag after the former had left. But I also commend you for taking the reins and trying to make the deadline.

But this is an experience I would like to have under my belt someday, albeit not an enjoyable one but just knowing what to do in that situation is alwys something you can take from it.

DPG

You will need to get used to the never-ending cycle of IT incompetence. Be assured that the next person will always blame all problems on the previous person.

Darthn3ss

Just out of curiosity, why will it take 30-60 days to fix?

Days like that are the reason why whiskey was invented.

Iristheangel

It takes 30-60 days to complete because it's like submitted a new order to the service provider. It's their timeframe we work with.

@DPG - I know the blame game but in this situation, it actually WAS the previous guy. I didn't actually place those orders and I was only supposed to ensure that the project was carried the rest of the way after he left. Theoretically, I could have combed through every order he placed, verified that the address was correct with the end users, and then corrected it a month ago (which still could have delayed things by a month because the SP would reset the clock on the order) so I can take some blame on that but I don't think, realistically, most of us would be backtracking through every detail of a project that was handed to you in it's last stage by a superior engineer. I think most of us would assume that senior engineer that was working on it had the basics correct and then make adjustments as we need to ensure SLA was met.

DPG

Iristheangel wrote: »

It takes 30-60 days to complete because it's like submitted a new order to the service provider. It's their timeframe we work with.

@DPG - I know the blame game but in this situation, it actually WAS the previous guy. I didn't actually place those orders and I was only supposed to ensure that the project was carried the rest of the way after he left. Theoretically, I could have combed through every order he placed, verified that the address was correct with the end users, and then corrected it a month ago (which still could have delayed things by a month because the SP would reset the clock on the order) so I can take some blame on that but I don't think, realistically, most of us would be backtracking through every detail of a project that was handed to you in it's last stage by a superior engineer. I think most of us would assume that senior engineer that was working on it had the basics correct and then make adjustments as we need to ensure SLA was met.

I completely understand that this isn't your fault. However, I have learned to assume NOTHING when it comes to these things.

it_consultant

Change orders with the ISP can easily be the most messed up part of your day. I have found myself, more than once, doing a little rain dance that the order is fulfilled properly and the demarc is extended without issue.

cgrimaldo

Wow..I have some had some rough days at work with vendors, CLECs, providers and construction folks but you have me beat. lol. Glad it is almost over and it's getting resolved!

bryguy

The worst is when it has nothing to do with you, and you have no means to fix it. I can't tell you how many 3:00 a.m. phone calls I received indicating a circuit was down and it was a backhoe, rat, fire, tornado, etc.

it_consultant

On our first freeze in Denver, we find out very fast how many of the southern CL techs they shipped up for summer work, none of their stuff is weatherproofed. Like clockwork, every November, at least 2 of my clients would lose their phones.

paulgswanson

Wow, that was a bad day. But you handled it like-a-boss. Many Kudos to you for the effort!

tprice5

Definitely should've emailed ole' boy and gave him the, "what the hell?"

WiseWun

What was the senior engineer's reaction when he screwed up with the address? There's no way he can rub the blame on you.

Iristheangel

@Wisewun - He quit so who knows... He can't really blame or not blame it on me since he's not there anymore

WiseWun

My bad, I thought he quit the project not the company. Working night shift as we speak, boy do I need some coffee!