Work last night

kryolla

Well I had a fun time last night at work. We lost both OC3 links to a remote site. We checked change management to see if any maintenance was being done and the telco provider was doing some work but just network impacting and not customer impacting so of course a flash was sent out and before you knew it there was a bridge up with management and about 3-4 other groups. We had the on call tech driving to the site. We were trying to see if the 6509 cat or the 7613 cat had a hardware issue or was it coming from telco. After much time trying to get circuit id's so we can login to cisco adm mux on both ends to see where the AIS was coming we found out it was from telco. So we called telco company and they said they have maintenance going on from midnight to 2 am and it will be customer impacting. LOL

Anybody else have an interesting story to tell?

GT-Rob

Sounds like most days at my work lol. Something with a large impact is down, and we spend more time letting people know about it, then actually fixing it.


And telcos are the worst.

DefendingNetworks

My days aren't that interesting, unfortunately. I wish they were. Yes I like those kind of days every now and then. My days are boring because they are filled with just surfing the web.

mikearama

Had a fun one yesterday myself.

We run an MPLS network between us at HQ and all our regional offices. Our site in Vancouver went offline shortly after I walked in at 9am yesterday. Toyota has a 4-hour SLA with our telco to keep the network running, and we pay dearly for it.

There's no one onsite who's technical in nature, so I'm left trying to walk an office person through a few basic troubleshooting steps... reboot the router, test the ups, do some pinging.

After a couple conference calls with Bell techs, it was determined that the wic card on the PE router out there was bad, and a tech would have to be sent to replace it.

Wouldn't you know it... we weren't the only company in British Columbia with issues yesterday, and the tech finally arrived at 7pm, swapped the card in under 10 minutes, and the circuit came right back up.

Got some overtime out of the deal, and get to listen to my manager ream out our telco account manager at some point today.

On a positive note, our ASA's arrived... can't wait to swap them in for the pix's (pixen? pixi?) we've been running for years. Oh yeah.

DefendingNetworks

Wish I got to mess with those kinds of things, especially Cisco in nature. The most I get to deal with, usually, is someone can't print. Gah! With my certs and degree I wish I could find something closer to home and a better job.

Once the housing market comes back, its bye bye Asheville.

Andretii

DefendingNetworks wrote:

My days aren't that interesting, unfortunately. I wish they were. Yes I like those kind of days every now and then. My days are boring because they are filled with just surfing the web.

I second that. or ??

DefendingNetworks

I really need to find another job. Working for an RV Dealership doesn't really further your career or expose you to newer technologies. Computer Science degree + CCNA and I'm stuck doing this. Its the area I live in, oh well, for now at least.

kryolla

In the cellular business we say who is our best drive testers? The customers LOL we were doing an upgrade on one of the cat and forgot a ACL entry and who found it out, of course the customers they were all complaining

Time for me to get some sleep

JohnDouglas

i spend much of my working life directly or indirectly causing unplanned outages while we migrate onto new equipment. keeps the support teams active

normally outages are due to remote support misreading my instructions. once had an engineer remove the wrong card. customers down, he drove 250 miles home to bed. got the call about the customer outage and had to head back again to reinsert the card. it's all overtime for them though.

sadly due to my massive workload now my mistakes are becoming more common (not helped by a poor network inventory). i think of it as collateral damage.

oh another one, had a tech take out a whole site once. he misunderstood (english as a second language) my instructions and performed a hot cutover one morning rather than preparing it. massive outage. happened just as the MD was apologising to a customer (on the customer site) for the late delivery of one of their circuits - they lost all connectivity. not a happy day. learned never to give verbal instructions that day.

GT-Rob

ugh, I feel bad for the techs that do the hardware installs/changes.


3:00am

*wakes up* "uhhh hello?"
"Hi there, sorry to wake you up, but we need a module repacement at X datacenter"
"um ok. Thats about 50 miles from here. When do you need it?"
"within an hour..."



Poor guys. I only hope they are getting paid a lot

Mishra

GT-Rob wrote:

ugh, I feel bad for the techs that do the hardware installs/changes.


3:00am

*wakes up* "uhhh hello?"
"Hi there, sorry to wake you up, but we need a module repacement at X datacenter"
"um ok. Thats about 50 miles from here. When do you need it?"
"within an hour..."



Poor guys. I only hope they are getting paid a lot

hehe agreed.

Fortunately most of them work 3rd shift.

However, when those big companies like GE demand a tech immediately even after the company saying it could wait until tomorrow, you bet your butt 2-3 people are waking up. There has to be a manager on the bridge making sure his tech gets to the place you know.

mikearama

Mishra wrote:

However, when those big companies like GE demand a tech immediately even after the company saying it could wait until tomorrow, you bet your butt 2-3 people are waking up. There has to be a manager on the bridge making sure his tech gets to the place you know.

+1

Like when I get woken up, it is to a bridge... and tier 2 support person and the desk manager are always on the line. At my gig, we're all contractors... as soon as I remotely connect, I'm billing 4 hours, so Operations exhausts every option first.

By the way, for you techies not at a big company, or still studying... a "bridge" is a conference call. I remember the first time I saw "bridge" and I thought... there's a manager on a bridge somewhere pointing in the right direction so his people don't get lost. D'oh! Course, every now and then when someone's pulled a beauty ID10T error, I wanna jump off a bridge.

Mmartin_47

Well my father works at a datacenter for an power company out here. He gets off work and it's a 30-40 minute drive home, everything is fine, then he is called back right when he comes home...

Johnny: "Hi can I speak to Verne, this is Johnny from PG&E"
My dad: "Hey Johnny, yeah I just reached home"
Johnny: "Our mainframe went offline and our SAP server cluster failed, can you come down?"
My dad: "you owe me a case of beer and a pizza"

Plazma

I freaking hate it when they do stuff like that and not tell anyone.. it makes me go into OMGWTF mode

DefendingNetworks

Plazma wrote:

I freaking hate it when they do stuff like that and not tell anyone.. it makes me go into OMGWTF mode

Much like your avatar on the left.

Plazma

DefendingNetworks wrote:

Plazma wrote:

I freaking hate it when they do stuff like that and not tell anyone.. it makes me go into OMGWTF mode

Much like your avatar on the left.

That's Jeremy when he was on cable back in the day ..

GET OFF MY BANDWIDTH!!!! ARRRGHGH!