Aw man, stupid Telecoms Engineer
jibbajabba
Google NinjaMember Posts: 4,317 ■■■■■■■■□□
in Off-Topic
Customer of ours ordered a leased line as a backup for his server.
Engineer turned up today with a router.
So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.
BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..
Some idiots should just be shot ...
Engineer turned up today with a router.
So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.
BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..
Some idiots should just be shot ...
My own knowledge base made public: http://open902.com 
Comments
You're absolutely right, some ppl should be shot!
-Bender
As our links to their equipment remained up, our network monitoring tools didn't notice anything. Our internet and VOIP calls originate from the co-lo site, but everything failed over so fast that nobody noticed anything. I saw more traffic than usual on the alternate link and found a little red light on their equipment. They reconnected the cirucit and everything went back to normal. I was actually excited at the chance to test the recoverability of our links. I did the design and implementation, but didn't have the guts (or approval) to pull a plug in the middle of the day just to see what would happen.
2 wire men come to office and install the new line.
The next day one guy come and connect the router.
on the third day another guy come and test the connection
if it does not work he calls the first 2 men to check the lines "one more day wasted"
they confirm that "there was a problem in the jumper and it is fixed now"
the testing guy come again and test, if it does not work call the wiring guys again ..
A password doesn't really prevent someone taking the router PHYSICALLY out of the cabinet
Wow ! This guy didn't have someone from your network team escorting him ?? I find that mind boggling !
I am a comms tech for an outsourcer and not a chance in hell I would let any carrier engineer loose in my comms rooms without me or a colleague being there. I am not paranoid or a control freak, they just don't know our network or how we use our wan circuits, nor do they care whether lines are live or not. Several times my being there has been needed and prevented major outages when they have said "I will just unplug this circuit to test it" without caring that I have several thousand users using that line. We even have the comms room seperated by a physical cage wall into carrier and our network with no structured patching in between. Hell, even only essential IT staff are allowed in there and only if they have a valid reason.
Really think you should take this very seriously and put policies in place. The client that got switched off has every rite to treat this as a major contract breach and rapidly change to a new company and sue your company for the costs of it all. Not to mention the bad press involved. God knows what this disconnection is costing them.
Realise I sound over the top and I appologise for not being supportive but this is outsourcing basics and should never have happened.
We had an engineer escorting him .. Well the engineer was from the datacenter and not from us. Well apparently we get the router back tomorrow 11am ... but yea - it will be interesting to see which SLA covers what ..
/slaps forehead
That guy is gonna be headed off to McDonalds.
I know of 4 DCs in the dlr area I think you are talking about. Could you PM me which ?
Its not a secret
We use two datacenter in the docklands area. Telehouse and Gyron - Gyron is in the Telstra building.
And tell me about McDonalds .. damn Datacenter lunches / dinners
I have seen this issue quite a few times at my last job at a very large telco. So many different databases to track circuit ids and they are all maintained by different groups. Sometimes the only way to tell if you were on the right circuit was to loop it to make sure the right customer went down
I'm working at a smaller ISP now so its not that big of an issue here.
/phew ... it's not one of mine then
Had me worried there for a sec /whistle
Lier - gimme the router back
So bottom line : It was agreed that the responsibility was solely the customer ones .. He ordered the engineer .. the engineer had a wrong work order .. so .. boffed
+1 We have a DC in the docklands area too and for a split second I thought Gomjaba and I worked for the same company hehe
Which would surely a good thing
But unless you have a German imported from Gibraltar working in your company - unlikely
Nah when I worked there it was common, mostly from the sales person who put an order in to remove such and such when they misunderstood the customer. Also the databases are huge and not properly administered with two many hands being in the cookie jar at all times. Another problem is with the plant outside, ex. Your buddy had a job on putting a t-1 in on cable 941-305 for the main and local 31-167 for the local, well 941-305 tests clean, but the local is bad with a hard short on it. It's hot 95 degrees that day or rainy at 33 degrees. He looks at 941-306 instead and and uses this pair as its vacant. He is in a hurry and calls loop assignment who forgets to update the records, well your buddy was in a hurry and he forgot to check to make sure the records where updated. You go out in three months and use that local pair going into the building. You were given 941-306 per the engineers and assignment so you use this pair. Well guess what 941-306 has a t-1 you just took out of service. Blame it on your buddy, but do you smack him for it, no you buy him a beer on the sweet double time call out you pulled that evening. You go out to subs, verify it there and then to the x-connect and check it there. You see it's not working on the correct pair and you clear it. I saw this all the time.
T-1 repair is a $100k+ job at the telco's. But it's not easy work as you are on poles and in manholes. The job is very techinical and physical at the same time.
Thas why i like labeling EVERYTHING
Bsc (hons) Network Computing - 1st Class
WIP: Msc advanced networking
Oh, we are labelled to max too
But like I said earlier - it turned out that both router belong to the same customer and therefore didn't raise a red flag with the onsite-engineer :P