Aw man, stupid Telecoms Engineer

jibbajabbajibbajabba Google NinjaMember Posts: 4,317 ■■■■■■■■□□
Customer of ours ordered a leased line as a backup for his server.
Engineer turned up today with a router.

So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

Some idiots should just be shot ...
My own knowledge base made public: http://open902.com :p

Comments

  • apd123apd123 Senior Member Member Posts: 171
    I will take your word but did this random Router not have a password on it?
  • tierstentiersten was here. Member Posts: 4,505
    Ouch. How exactly do you explain that to the other customer? Sorry... an engineer accidentally stole your router...
  • AldurAldur Juniper Moderator Member Posts: 1,460
    haha wow, what goes though somebody's head to make them think it's ok to pull a working router and put it somewhere else without knowing what that router does first....

    You're absolutely right, some ppl should be shot!icon_surprised.gif
    "Bribe is such an ugly word. I prefer extortion. The X makes it sound cool."

    -Bender
  • ClaymooreClaymoore Nidhoggr, the Net Serpent Member Posts: 1,637
    At my previous job our provider accidentally disconnected our OC3 circuit between our main site and our co-lo site. We made a change to the circuit 6 months earlier in order to move to an MPLS ethernet handoff and that change had to be proccessed as a disconnect/reconnect order (it was a paperwork thing). They made the 'reconnect' part of the change as quickly as a large telco typically does but they didn't get around to the 'disconnect' part of the change until 6 months later. Problem is the circuit was live - just reconfigured - but they disconnected anyway. Fortunately we had an alternate provider and EIGRP.

    As our links to their equipment remained up, our network monitoring tools didn't notice anything. Our internet and VOIP calls originate from the co-lo site, but everything failed over so fast that nobody noticed anything. I saw more traffic than usual on the alternate link and found a little red light on their equipment. They reconnected the cirucit and everything went back to normal. I was actually excited at the chance to test the recoverability of our links. I did the design and implementation, but didn't have the guts (or approval) to pull a plug in the middle of the day just to see what would happen.
  • wd40wd40 Member Posts: 1,017 ■■■■□□□□□□
    It goes like this here:

    2 wire men come to office and install the new line.

    The next day one guy come and connect the router.

    on the third day another guy come and test the connection :D

    if it does not work he calls the first 2 men to check the lines "one more day wasted"

    they confirm that "there was a problem in the jumper and it is fixed now"

    the testing guy come again and test, if it does not work call the wiring guys again ..
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    apd123 wrote: »
    I will take your word but did this random Router not have a password on it?

    A password doesn't really prevent someone taking the router PHYSICALLY out of the cabinet :)
    My own knowledge base made public: http://open902.com :p
  • KaminskyKaminsky Senior Member Member Posts: 1,235
    Gomjaba wrote: »
    Customer of ours ordered a leased line as a backup for his server.
    Engineer turned up today with a router.

    So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

    BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

    Some idiots should just be shot ...


    Wow ! This guy didn't have someone from your network team escorting him ?? I find that mind boggling !

    I am a comms tech for an outsourcer and not a chance in hell I would let any carrier engineer loose in my comms rooms without me or a colleague being there. I am not paranoid or a control freak, they just don't know our network or how we use our wan circuits, nor do they care whether lines are live or not. Several times my being there has been needed and prevented major outages when they have said "I will just unplug this circuit to test it" without caring that I have several thousand users using that line. We even have the comms room seperated by a physical cage wall into carrier and our network with no structured patching in between. Hell, even only essential IT staff are allowed in there and only if they have a valid reason.

    Really think you should take this very seriously and put policies in place. The client that got switched off has every rite to treat this as a major contract breach and rapidly change to a new company and sue your company for the costs of it all. Not to mention the bad press involved. God knows what this disconnection is costing them.

    Realise I sound over the top and I appologise for not being supportive but this is outsourcing basics and should never have happened.
    Kam.
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    Kaminsky wrote: »
    Wow ! This guy didn't have someone from your network team escorting him ?? I find that mind boggling !

    We had an engineer escorting him .. Well the engineer was from the datacenter and not from us. Well apparently we get the router back tomorrow 11am ... but yea - it will be interesting to see which SLA covers what ..
    My own knowledge base made public: http://open902.com :p
  • KaminskyKaminsky Senior Member Member Posts: 1,235
    Gomjaba wrote: »
    We had an engineer escorting him .. Well the engineer was from the datacenter and not from us. Well apparently we get the router back tomorrow 11am ... but yea - it will be interesting to see which SLA covers what ..


    /slaps forehead

    That guy is gonna be headed off to McDonalds.

    I know of 4 DCs in the dlr area I think you are talking about. Could you PM me which ?
    Kam.
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    Kaminsky wrote: »
    /slaps forehead

    That guy is gonna be headed off to McDonalds.

    I know of 4 DCs in the dlr area I think you are talking about. Could you PM me which ?

    Its not a secret :)

    We use two datacenter in the docklands area. Telehouse and Gyron - Gyron is in the Telstra building.

    And tell me about McDonalds .. damn Datacenter lunches / dinners :)
    My own knowledge base made public: http://open902.com :p
  • msteinhilbermsteinhilber Senior Member Member Posts: 1,480 ■■■■■■■■□□
    One of the telco's we use decided one day to change a good chunk of our circuit ID's for no apparent reason. The problem was they have multiple systems in which we exist as a customer (TDS Telecom & TDS Metrocom). Somehow they managed to waste 3 days to bring a circuit back up because they ended up troubleshooting the wrong circuit ID (one for another business at another location) even though all of the dispatched technicians arrived at the proper address. It finally took the owner of our organization to get in touch with one of their VP's that he knew through family and then things started to actually move along. Prior to that it was a whole lot of fighting internally on their part (dispatched techs saying uhh yea, circuit down while the CO techs were saying no it's up) and yelling from our end.
  • networker050184networker050184 Went to the dark side.... Mod Posts: 11,962 Mod
    One of the telco's we use decided one day to change a good chunk of our circuit ID's for no apparent reason. The problem was they have multiple systems in which we exist as a customer (TDS Telecom & TDS Metrocom). Somehow they managed to waste 3 days to bring a circuit back up because they ended up troubleshooting the wrong circuit ID (one for another business at another location) even though all of the dispatched technicians arrived at the proper address. It finally took the owner of our organization to get in touch with one of their VP's that he knew through family and then things started to actually move along. Prior to that it was a whole lot of fighting internally on their part (dispatched techs saying uhh yea, circuit down while the CO techs were saying no it's up) and yelling from our end.

    I have seen this issue quite a few times at my last job at a very large telco. So many different databases to track circuit ids and they are all maintained by different groups. Sometimes the only way to tell if you were on the right circuit was to loop it to make sure the right customer went down :D

    I'm working at a smaller ISP now so its not that big of an issue here.
    An expert is a man who has made all the mistakes which can be made.
  • KaminskyKaminsky Senior Member Member Posts: 1,235
    Gomjaba wrote: »
    Its not a secret :)

    We use two datacenter in the docklands area. Telehouse and Gyron - Gyron is in the Telstra building.

    And tell me about McDonalds .. damn Datacenter lunches / dinners :)

    /phew ... it's not one of mine then :)

    Had me worried there for a sec /whistle
    Kam.
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    Kaminsky wrote: »
    /phew ... it's not one of mine then :)

    Lier - gimme the router back :p
    My own knowledge base made public: http://open902.com :p
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    Aaahhh .. found out why the customer wasn't so upset after all... Those the one which was taken was actually his as well .. so both router belonged to the customer. We then checked the work order of the engineer and it clearly stated to swap rather than add an additional one.

    So bottom line : It was agreed that the responsibility was solely the customer ones .. He ordered the engineer .. the engineer had a wrong work order .. so .. boffed :)
    My own knowledge base made public: http://open902.com :p
  • bertiebbertieb wibble! Member Posts: 1,031 ■■■■■■□□□□
    Kaminsky wrote: »
    /phew ... it's not one of mine then :)

    Had me worried there for a sec /whistle

    +1 We have a DC in the docklands area too and for a split second I thought Gomjaba and I worked for the same company hehe
    The trouble with quotes on the internet is that you can never tell if they are genuine - Abraham Lincoln
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    bertieb wrote: »
    +1 We have a DC in the docklands area too and for a split second I thought Gomjaba and I worked for the same company hehe

    Which would surely a good thing :):)

    But unless you have a German imported from Gibraltar working in your company - unlikely :):)
    My own knowledge base made public: http://open902.com :p
  • bellheadbellhead Senior Member Member Posts: 120
    Not to defend the telco's but they are very large organizations who rarely make mistakes..icon_redface.gif

    Nah when I worked there it was common, mostly from the sales person who put an order in to remove such and such when they misunderstood the customer. Also the databases are huge and not properly administered with two many hands being in the cookie jar at all times. Another problem is with the plant outside, ex. Your buddy had a job on putting a t-1 in on cable 941-305 for the main and local 31-167 for the local, well 941-305 tests clean, but the local is bad with a hard short on it. It's hot 95 degrees that day or rainy at 33 degrees. He looks at 941-306 instead and and uses this pair as its vacant. He is in a hurry and calls loop assignment who forgets to update the records, well your buddy was in a hurry and he forgot to check to make sure the records where updated. You go out in three months and use that local pair going into the building. You were given 941-306 per the engineers and assignment so you use this pair. Well guess what 941-306 has a t-1 you just took out of service. Blame it on your buddy, but do you smack him for it, no you buy him a beer on the sweet double time call out you pulled that evening. You go out to subs, verify it there and then to the x-connect and check it there. You see it's not working on the correct pair and you clear it. I saw this all the time.

    T-1 repair is a $100k+ job at the telco's. But it's not easy work as you are on poles and in manholes. The job is very techinical and physical at the same time.
  • nelnel Senior Member Member Posts: 2,859 ■□□□□□□□□□
    Gomjaba wrote: »
    Customer of ours ordered a leased line as a backup for his server.
    Engineer turned up today with a router.

    So what did that idiot do, he takes another router out of the cabinet, installs the new router and leaves.

    BUT: That router belonged to another customer - who is now offline as a result and the company cannot reach the engineer ..

    Some idiots should just be shot ...

    Thas why i like labeling EVERYTHING :D
    Xbox Live: Bring It On

    Bsc (hons) Network Computing - 1st Class
    WIP: Msc advanced networking
  • jibbajabbajibbajabba Google Ninja Member Posts: 4,317 ■■■■■■■■□□
    nel wrote: »
    Thas why i like labeling EVERYTHING :D

    Oh, we are labelled to max too :)

    But like I said earlier - it turned out that both router belong to the same customer and therefore didn't raise a red flag with the onsite-engineer :P
    My own knowledge base made public: http://open902.com :p
Sign In or Register to comment.