Options
Definition of a bad day...
Iristheangel
Mod Posts: 4,133 Mod
This is mostly just a rant because I've had a HORRIBLE day today and I wish I was nursing a beer while writing this but I'm sadly still working. I just want to preface this by saying we all have bad days and I love my job so I wouldn't trade my job because of a day like this. I'm definitely changing or switching around any potentially identifiable information in this post to protect the innocent
Today I was working on a new site build-out that I've been working on for the last month. I took it over from a senior network engineer who had quit mid-project. When this thing was handed to me, the engineer had ordered wiring for the office, MPLS, and redundant DSL and procured a $25K quote for network equipment for this tiny little temporary office that would be shutting down in 6 months. He hadn't spoken to the end-users or management about the needs and it was a bit of a cluster when it got to me. Very important: The only things I knew about this site were provided to me by him and his notes.
I spoke to the managers at the time, reassessed their wiring, equipment, and data needs and ended up with a quote for about $10K worth of equipment and changed the MPLS line to dual T1 lines since the install time would have been a few months longer for the MPLS to be installed (bureaucratic service provider and budget issues). I called up the service provider at the time and asked him to make the adjustments on the orders for DSL and the T1s but I did not change ANY other information provided by the Sr Engineer because he knew more about the site than I did. I ordered a nice little 2921 router with a PoE switchport module and some VoIP phones. When I reviewed the quote, I noted that the PoE power supply was not on it and specifically e-mailed the vendor to ask about it. He put my worries to rest by insisting it was included in the price.
Fast forward to a week ago - I get an e-mail from the service provider saying the T1 circuit and DSL is installed and ready to be turned on. While I have the VoIP phones and all the modules for the router, I do not have the router. I put pressure on the hardware vendor and he FINALLY finds out what was causing the delay and rushed it to be delivered to me. I received it on Tuesday with the growing pressure of knowing that the employees are going to be moving into this site on 11/15. I needed to hustle to configure this router for voice, routing and switching but I got hit with a major outage yesterday that kept all of us engineers busy until late at night.
Cue to this morning - First thing this morning, I get a call from our construction team manager who is surprised that the service provder didn't let him know the circuit was installed even though he was the contact. The reason being is that while he did run a line from the MPOE to the office, he did not connect it to anything because the SP was supposed to call him. It's going to take some time for him to get his tech out there to connect it and I'm running late configuring the router. While I'm working on the router, I can't get the switchport module to power ANYTHING over ethernet. About four engineers and I toiled on it for a good 3 hours only to realize that the vendor didn't send it with a PoE power supply. I end up having to run down 5 flights of stairs to our storage room, procure a switch from another project to use for this one temporarily, and configure it on the fly. After that, I FINALLY get everything in my car and run over to the site since it was close enough to our corporate office.
I'm onsite and connecting everything but we still don't have the tech onsite. After about 45 minutes, he finally shows up and goes to the MPOE to extend the line to the T1. After about 20 minutes, I get a call saying they cant find the circuit. I give him the circuit information and went back to my work. 30 minutes later, I get another call saying that they still can't find the circuit. After spending the next 5 hours going back and forth with the LEC, SP and onsite contact, we find out the problem: The original senior network engineer submitted both the DSL and T1 orders with VASTLY different addresses than the office's building's address (it's located in something else so it's very easy to do and very complicated)
So where we at tonight? ETA to fix this is 30-60 days. I had to explain to the CTO and IT director what exactly happened when I got to the office at 7pm and we're going to have to announce to about 12 employees that they cannot move into their new office for a couple more months. THIS is a bad day.
So... I'm not in any trouble and no one is blaming me but it was a cluster today. I still love my job and what I do. This was just the perfect storm of crap today.
/end rant
Today I was working on a new site build-out that I've been working on for the last month. I took it over from a senior network engineer who had quit mid-project. When this thing was handed to me, the engineer had ordered wiring for the office, MPLS, and redundant DSL and procured a $25K quote for network equipment for this tiny little temporary office that would be shutting down in 6 months. He hadn't spoken to the end-users or management about the needs and it was a bit of a cluster when it got to me. Very important: The only things I knew about this site were provided to me by him and his notes.
I spoke to the managers at the time, reassessed their wiring, equipment, and data needs and ended up with a quote for about $10K worth of equipment and changed the MPLS line to dual T1 lines since the install time would have been a few months longer for the MPLS to be installed (bureaucratic service provider and budget issues). I called up the service provider at the time and asked him to make the adjustments on the orders for DSL and the T1s but I did not change ANY other information provided by the Sr Engineer because he knew more about the site than I did. I ordered a nice little 2921 router with a PoE switchport module and some VoIP phones. When I reviewed the quote, I noted that the PoE power supply was not on it and specifically e-mailed the vendor to ask about it. He put my worries to rest by insisting it was included in the price.
Fast forward to a week ago - I get an e-mail from the service provider saying the T1 circuit and DSL is installed and ready to be turned on. While I have the VoIP phones and all the modules for the router, I do not have the router. I put pressure on the hardware vendor and he FINALLY finds out what was causing the delay and rushed it to be delivered to me. I received it on Tuesday with the growing pressure of knowing that the employees are going to be moving into this site on 11/15. I needed to hustle to configure this router for voice, routing and switching but I got hit with a major outage yesterday that kept all of us engineers busy until late at night.
Cue to this morning - First thing this morning, I get a call from our construction team manager who is surprised that the service provder didn't let him know the circuit was installed even though he was the contact. The reason being is that while he did run a line from the MPOE to the office, he did not connect it to anything because the SP was supposed to call him. It's going to take some time for him to get his tech out there to connect it and I'm running late configuring the router. While I'm working on the router, I can't get the switchport module to power ANYTHING over ethernet. About four engineers and I toiled on it for a good 3 hours only to realize that the vendor didn't send it with a PoE power supply. I end up having to run down 5 flights of stairs to our storage room, procure a switch from another project to use for this one temporarily, and configure it on the fly. After that, I FINALLY get everything in my car and run over to the site since it was close enough to our corporate office.
I'm onsite and connecting everything but we still don't have the tech onsite. After about 45 minutes, he finally shows up and goes to the MPOE to extend the line to the T1. After about 20 minutes, I get a call saying they cant find the circuit. I give him the circuit information and went back to my work. 30 minutes later, I get another call saying that they still can't find the circuit. After spending the next 5 hours going back and forth with the LEC, SP and onsite contact, we find out the problem: The original senior network engineer submitted both the DSL and T1 orders with VASTLY different addresses than the office's building's address (it's located in something else so it's very easy to do and very complicated)
So where we at tonight? ETA to fix this is 30-60 days. I had to explain to the CTO and IT director what exactly happened when I got to the office at 7pm and we're going to have to announce to about 12 employees that they cannot move into their new office for a couple more months. THIS is a bad day.
So... I'm not in any trouble and no one is blaming me but it was a cluster today. I still love my job and what I do. This was just the perfect storm of crap today.
/end rant
Comments
-
OptionsDPG Member Posts: 780 ■■■■■□□□□□I've definitely had plenty of these type of days working in IT. They usually make you want to quit and never have to deal with another electronic device again. At least there are usually way more positive days in which you learn something cool or get to save the company's ass.
-
Optionsemerald_octane Member Posts: 613It looks like you handled it with class. This may end up looking better for you in the long term because you deal with these massive issues as they come. I wonder if the old engineer QUIT or was he FIRED. Sounds pretty incompetent.
If it's any consolation, i've been in hot water with my CEO over why I couldn't convince the phone company and comcast to move our T1 and CATV lines....over a HOLIDAY weekend (4th of july, which I missed because he wanted no downtime). Don't forget we are members of (ISC)2. In theory, we can handle anything LOL. -
Optionsdemonfurbie Member Posts: 1,819welcome to network design and dealing with subs
the easy part is config/design the hard part is dealing with the "others" (contractors, users, service providers, ect ect ect)
i have gotten to the point where i ask for blueprints of the building before i start any kinda planning and i have the all the equipment delivered to me (not wiring ... duhhh) and i go plug it in my self.
and when dealing with service providers and contractors make sure you get a meeting with them first via phone or in person so they can all be on the same page with what you have planned, this also helps iron out any issues that both may havewgu undergrad: done ... woot!!
WGU MS IT Management: done ... double woot :cheers: -
OptionsIristheangel Mod Posts: 4,133 ModWe're an international organization so plugging it in for myself in every site doesn't work. Usually I have the equipment shipped to me and sent onsite but this was a special case that happened to be so close and we were approaching the deadline, I went onsite for this one time. Anyways, it was a cluster... I didn't start this project, the original specs were not correct and there were all sorts of delays before I could actually order the equipment (budget squabbles - the original $25K quote ruffled a lot of feathers)
-
OptionsKaii Member Posts: 59 ■■□□□□□□□□truly bad day but still love your job .
happy to hear it ^_^I`m the Designer of my Own Catastrophy
“The best way to predict the FUTURE is to CREATE it” -
Optionsm3zilla Member Posts: 172I've definitely had plenty of these type of days working in IT. They usually make you want to quit and never have to deal with another electronic device again.
The worse part is when you realized what happened, and you wonder how you could have missed the issue, or screwed up so badly! -
OptionsNotHackingYou Member Posts: 1,460 ■■■■■■■■□□Sounds like a bad one!When you go the extra mile, there's no traffic.
-
Optionstruckfit Banned Posts: 78 ■■□□□□□□□□you survived
what doesn't kill you make you stronger.
Nice one -
Optionsuyen_nguyen Member Posts: 32 ■■□□□□□□□□A bad day is when teachers give unexpected pop quiz.English is my second language. My apology for my grammar errors.
-
Optionsboredgamelad Member Posts: 365 ■■■■□□□□□□It wouldn't be a pop quiz if you were expecting it, now would it?Iristheangel wrote: »So... I'm not in any trouble and no one is blaming me but it was a cluster today. I still love my job and what I do. This was just the perfect storm of crap today.
Thanks for sharing the story. As crappy as that day was, it sounds like you're doing exactly the kind of work I'd like to be doing someday. -
Optionsstlsmoore Member Posts: 515 ■■■□□□□□□□Yea rough days can be stressful.
Today for me lost another module on our core nexus switch again for the second time within two weeks. Just found out the rack that this device is in is currently oversubscribed on power, it's currently at 110%! Think I found why this issue keeps happening at least, on hold now with TAC to verify.My Cisco Blog Adventure: http://shawnmoorecisco.blogspot.com/
Don't Forget to Add me on LinkedIn!
https://www.linkedin.com/in/shawnrmoore -
OptionsRoguetadhg Member Posts: 2,489 ■■■■■■■■□□Double Facepalm.
My bad days revolve around: "Tomorrow we have a client coming in. We need to have 40 booths setup, imaged, and ready to go."In order to succeed, your desire for success should be greater than your fear of failure.
TE Threads: How to study for the CCENT/CCNA, Introduction to Cisco Exams -
OptionsRomBUS Member Posts: 699 ■■■■□□□□□□Wow sucks that you were left holding the bag after the former had left. But I also commend you for taking the reins and trying to make the deadline.
But this is an experience I would like to have under my belt someday, albeit not an enjoyable one but just knowing what to do in that situation is alwys something you can take from it. -
OptionsDPG Member Posts: 780 ■■■■■□□□□□You will need to get used to the never-ending cycle of IT incompetence. Be assured that the next person will always blame all problems on the previous person.
-
OptionsDarthn3ss Member Posts: 1,096Just out of curiosity, why will it take 30-60 days to fix?
Days like that are the reason why whiskey was invented.Fantastic. The project manager is inspired.
In Progress: 70-640, 70-685 -
OptionsIristheangel Mod Posts: 4,133 ModIt takes 30-60 days to complete because it's like submitted a new order to the service provider. It's their timeframe we work with.
@DPG - I know the blame game but in this situation, it actually WAS the previous guy. I didn't actually place those orders and I was only supposed to ensure that the project was carried the rest of the way after he left. Theoretically, I could have combed through every order he placed, verified that the address was correct with the end users, and then corrected it a month ago (which still could have delayed things by a month because the SP would reset the clock on the order) so I can take some blame on that but I don't think, realistically, most of us would be backtracking through every detail of a project that was handed to you in it's last stage by a superior engineer. I think most of us would assume that senior engineer that was working on it had the basics correct and then make adjustments as we need to ensure SLA was met. -
OptionsDPG Member Posts: 780 ■■■■■□□□□□Iristheangel wrote: »It takes 30-60 days to complete because it's like submitted a new order to the service provider. It's their timeframe we work with.
@DPG - I know the blame game but in this situation, it actually WAS the previous guy. I didn't actually place those orders and I was only supposed to ensure that the project was carried the rest of the way after he left. Theoretically, I could have combed through every order he placed, verified that the address was correct with the end users, and then corrected it a month ago (which still could have delayed things by a month because the SP would reset the clock on the order) so I can take some blame on that but I don't think, realistically, most of us would be backtracking through every detail of a project that was handed to you in it's last stage by a superior engineer. I think most of us would assume that senior engineer that was working on it had the basics correct and then make adjustments as we need to ensure SLA was met.
I completely understand that this isn't your fault. However, I have learned to assume NOTHING when it comes to these things. -
Optionsit_consultant Member Posts: 1,903Change orders with the ISP can easily be the most messed up part of your day. I have found myself, more than once, doing a little rain dance that the order is fulfilled properly and the demarc is extended without issue.
-
Optionscgrimaldo Member Posts: 439 ■■■■□□□□□□Wow..I have some had some rough days at work with vendors, CLECs, providers and construction folks but you have me beat. lol. Glad it is almost over and it's getting resolved!
-
Optionsbryguy Member Posts: 190The worst is when it has nothing to do with you, and you have no means to fix it. I can't tell you how many 3:00 a.m. phone calls I received indicating a circuit was down and it was a backhoe, rat, fire, tornado, etc.
-
Optionsit_consultant Member Posts: 1,903On our first freeze in Denver, we find out very fast how many of the southern CL techs they shipped up for summer work, none of their stuff is weatherproofed. Like clockwork, every November, at least 2 of my clients would lose their phones.
-
Optionspaulgswanson Member Posts: 311Wow, that was a bad day. But you handled it like-a-boss. Many Kudos to you for the effort!http://paulswansonblog.wordpress.com/
WGU Progress: B.S. Network Management & Design <- I quit (got bored) -
Optionstprice5 Member Posts: 770Definitely should've emailed ole' boy and gave him the, "what the hell?"Certification To-Do: CEH [ ], CHFI [ ], NCSA [ ], E10-001 [ ], 70-413 [ ], 70-414 [ ]
WGU MSISA
Start Date: 10/01/2014 | Complete Date: ASAP
All Courses: LOT2, LYT2 , UVC2, ORA1, VUT2, VLT2 , FNV2 , TFT2 , JIT2 , FMV2, FXT2 , LQT2 -
OptionsWiseWun Member Posts: 285What was the senior engineer's reaction when he screwed up with the address? There's no way he can rub the blame on you."If you’re not prepared to be wrong, you’ll never come up with anything original.” - Ken Robinson
-
OptionsIristheangel Mod Posts: 4,133 Mod@Wisewun - He quit so who knows... He can't really blame or not blame it on me since he's not there anymore
-
OptionsWiseWun Member Posts: 285My bad, I thought he quit the project not the company. Working night shift as we speak, boy do I need some coffee!"If you’re not prepared to be wrong, you’ll never come up with anything original.” - Ken Robinson