Aw man, stupid Telecoms Engineer

jibbajabba

Customer of ours ordered a leased line as a backup for his server.
Engineer turned up today with a router.

So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

Some idiots should just be shot ...

apd123

I will take your word but did this random Router not have a password on it?

tiersten

Ouch. How exactly do you explain that to the other customer? Sorry... an engineer accidentally stole your router...

Aldur

haha wow, what goes though somebody's head to make them think it's ok to pull a working router and put it somewhere else without knowing what that router does first....

You're absolutely right, some ppl should be shot!

Claymoore

At my previous job our provider accidentally disconnected our OC3 circuit between our main site and our co-lo site. We made a change to the circuit 6 months earlier in order to move to an MPLS ethernet handoff and that change had to be proccessed as a disconnect/reconnect order (it was a paperwork thing). They made the 'reconnect' part of the change as quickly as a large telco typically does but they didn't get around to the 'disconnect' part of the change until 6 months later. Problem is the circuit was live - just reconfigured - but they disconnected anyway. Fortunately we had an alternate provider and EIGRP.

As our links to their equipment remained up, our network monitoring tools didn't notice anything. Our internet and VOIP calls originate from the co-lo site, but everything failed over so fast that nobody noticed anything. I saw more traffic than usual on the alternate link and found a little red light on their equipment. They reconnected the cirucit and everything went back to normal. I was actually excited at the chance to test the recoverability of our links. I did the design and implementation, but didn't have the guts (or approval) to pull a plug in the middle of the day just to see what would happen.

wd40

It goes like this here:

2 wire men come to office and install the new line.

The next day one guy come and connect the router.

on the third day another guy come and test the connection

if it does not work he calls the first 2 men to check the lines "one more day wasted"

they confirm that "there was a problem in the jumper and it is fixed now"

the testing guy come again and test, if it does not work call the wiring guys again ..

jibbajabba

apd123 wrote: »

I will take your word but did this random Router not have a password on it?

A password doesn't really prevent someone taking the router PHYSICALLY out of the cabinet

Kaminsky

Gomjaba wrote: »

Customer of ours ordered a leased line as a backup for his server.
Engineer turned up today with a router.

So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

Some idiots should just be shot ...

Wow ! This guy didn't have someone from your network team escorting him ?? I find that mind boggling !

I am a comms tech for an outsourcer and not a chance in hell I would let any carrier engineer loose in my comms rooms without me or a colleague being there. I am not paranoid or a control freak, they just don't know our network or how we use our wan circuits, nor do they care whether lines are live or not. Several times my being there has been needed and prevented major outages when they have said "I will just unplug this circuit to test it" without caring that I have several thousand users using that line. We even have the comms room seperated by a physical cage wall into carrier and our network with no structured patching in between. Hell, even only essential IT staff are allowed in there and only if they have a valid reason.

Really think you should take this very seriously and put policies in place. The client that got switched off has every rite to treat this as a major contract breach and rapidly change to a new company and sue your company for the costs of it all. Not to mention the bad press involved. God knows what this disconnection is costing them.

Realise I sound over the top and I appologise for not being supportive but this is outsourcing basics and should never have happened.

jibbajabba

Kaminsky wrote: »

Wow ! This guy didn't have someone from your network team escorting him ?? I find that mind boggling !

We had an engineer escorting him .. Well the engineer was from the datacenter and not from us. Well apparently we get the router back tomorrow 11am ... but yea - it will be interesting to see which SLA covers what ..

Kaminsky

Gomjaba wrote: »

We had an engineer escorting him .. Well the engineer was from the datacenter and not from us. Well apparently we get the router back tomorrow 11am ... but yea - it will be interesting to see which SLA covers what ..

/slaps forehead

That guy is gonna be headed off to McDonalds.

I know of 4 DCs in the dlr area I think you are talking about. Could you PM me which ?

jibbajabba

Kaminsky wrote: »

/slaps forehead

That guy is gonna be headed off to McDonalds.

I know of 4 DCs in the dlr area I think you are talking about. Could you PM me which ?

Its not a secret

We use two datacenter in the docklands area. Telehouse and Gyron - Gyron is in the Telstra building.

And tell me about McDonalds .. damn Datacenter lunches / dinners

msteinhilber

One of the telco's we use decided one day to change a good chunk of our circuit ID's for no apparent reason. The problem was they have multiple systems in which we exist as a customer (TDS Telecom & TDS Metrocom). Somehow they managed to waste 3 days to bring a circuit back up because they ended up troubleshooting the wrong circuit ID (one for another business at another location) even though all of the dispatched technicians arrived at the proper address. It finally took the owner of our organization to get in touch with one of their VP's that he knew through family and then things started to actually move along. Prior to that it was a whole lot of fighting internally on their part (dispatched techs saying uhh yea, circuit down while the CO techs were saying no it's up) and yelling from our end.

networker050184

msteinhilber wrote: »

One of the telco's we use decided one day to change a good chunk of our circuit ID's for no apparent reason. The problem was they have multiple systems in which we exist as a customer (TDS Telecom & TDS Metrocom). Somehow they managed to waste 3 days to bring a circuit back up because they ended up troubleshooting the wrong circuit ID (one for another business at another location) even though all of the dispatched technicians arrived at the proper address. It finally took the owner of our organization to get in touch with one of their VP's that he knew through family and then things started to actually move along. Prior to that it was a whole lot of fighting internally on their part (dispatched techs saying uhh yea, circuit down while the CO techs were saying no it's up) and yelling from our end.

I have seen this issue quite a few times at my last job at a very large telco. So many different databases to track circuit ids and they are all maintained by different groups. Sometimes the only way to tell if you were on the right circuit was to loop it to make sure the right customer went down

I'm working at a smaller ISP now so its not that big of an issue here.

Kaminsky

Gomjaba wrote: »

Its not a secret

We use two datacenter in the docklands area. Telehouse and Gyron - Gyron is in the Telstra building.

And tell me about McDonalds .. damn Datacenter lunches / dinners

/phew ... it's not one of mine then

Had me worried there for a sec /whistle

jibbajabba

Kaminsky wrote: »

/phew ... it's not one of mine then

Lier - gimme the router back

jibbajabba

Aaahhh .. found out why the customer wasn't so upset after all... Those the one which was taken was actually his as well .. so both router belonged to the customer. We then checked the work order of the engineer and it clearly stated to swap rather than add an additional one.

So bottom line : It was agreed that the responsibility was solely the customer ones .. He ordered the engineer .. the engineer had a wrong work order .. so .. boffed

bertieb

Kaminsky wrote: »

/phew ... it's not one of mine then

Had me worried there for a sec /whistle

+1 We have a DC in the docklands area too and for a split second I thought Gomjaba and I worked for the same company hehe

jibbajabba

bertieb wrote: »

+1 We have a DC in the docklands area too and for a split second I thought Gomjaba and I worked for the same company hehe

Which would surely a good thing

But unless you have a German imported from Gibraltar working in your company - unlikely :)

bellhead

Not to defend the telco's but they are very large organizations who rarely make mistakes..

Nah when I worked there it was common, mostly from the sales person who put an order in to remove such and such when they misunderstood the customer. Also the databases are huge and not properly administered with two many hands being in the cookie jar at all times. Another problem is with the plant outside, ex. Your buddy had a job on putting a t-1 in on cable 941-305 for the main and local 31-167 for the local, well 941-305 tests clean, but the local is bad with a hard short on it. It's hot 95 degrees that day or rainy at 33 degrees. He looks at 941-306 instead and and uses this pair as its vacant. He is in a hurry and calls loop assignment who forgets to update the records, well your buddy was in a hurry and he forgot to check to make sure the records where updated. You go out in three months and use that local pair going into the building. You were given 941-306 per the engineers and assignment so you use this pair. Well guess what 941-306 has a t-1 you just took out of service. Blame it on your buddy, but do you smack him for it, no you buy him a beer on the sweet double time call out you pulled that evening. You go out to subs, verify it there and then to the x-connect and check it there. You see it's not working on the correct pair and you clear it. I saw this all the time.

T-1 repair is a $100k+ job at the telco's. But it's not easy work as you are on poles and in manholes. The job is very techinical and physical at the same time.

nel

Gomjaba wrote: »

Customer of ours ordered a leased line as a backup for his server.
Engineer turned up today with a router.

So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

Some idiots should just be shot ...

Thas why i like labeling EVERYTHING

jibbajabba

nel wrote: »

Thas why i like labeling EVERYTHING

Oh, we are labelled to max too

But like I said earlier - it turned out that both router belong to the same customer and therefore didn't raise a red flag with the onsite-engineer :P