Epic Fails

NightShade1

if you have any history of an epic fail or funny history this is where you post it heh

okay ill start
The other day i installed a firewall on a client, this client has like i dont know could be like 50 small branches so its not that small...

Anyways i install my firewall i configure it and im done

like 3 days after i get a call of the it manager telling me that there was something wrong with the firewall... they didnt have email.... and he wanted me there now! i tell my boss that i could help him to troubleshoot from the office and well he told me that if he wants to pay for the visit then i should go, and he tell me to go.
I went in there i sit in the chair in a random computer i did ping Google and i get a ping reply so internet okay
i opened fast an outllook, and looked for their pop3 server and smtp to see the which was it
i try pinging the name and it coudlnt not resolve it
so i said weird...
i do ipconfig -all
and i get notice they dont have internal DNS server O_o they point it to the Carrier DNS(Fail)
As soon i did this, one of the techsupport guy tell me that there was one computer that did work the email so i tell him well show me the computer that works
i look at the outlook config and see it had the IP address instead the name on the pop3 and smtp

so i said well heheheh i went back to the same comptuer iw as working before and i entered in the config of the network card changed the DNS to 4.2.2.1 and worked

The issue was the carrier DNS that were Fuc*** up

And this was FAIL
i even didnt enter the firewall management console neither the cli to solve this issue...
it took me around 10 minutes to solve it and i think my boss charge him a lot of money because it was not even the firewall and i had to fix something that didnt had to do with us..

andlike this there are many... the IT deparment isnt really good....


Here is another one

I went to install a cluster of firewalls in a client and configure it... he had many vlans and everything but the administrator wasnt soo sure of the vlans that were in there and i was helping him with his switch to fireugre out which vlans he had... i had to check by myself O_o
okay this wasnt a good sign.. anyways i finished my job and then again
like 5 days after the installation the client call us and my boss get the call,( i was on a cleint while this was happening)
he was telling my boss that the internet was super slow and it was the firewall fault...
my boss connect to the firewall and indeed the BW usage was at the top....
so my boss start to troubleshoot it, and he figure it out that a camara he had in there was consuming a lot of BW.... so he does some traffic shapping on it...
the BW normalizeand he send a screenshot of the BW usage to the client
the client call back like an hour after it(i was in the office thsi time) and tells my boss its still super slow! its the firewall! he said. and my boss was telling hiim that waswerid the bw to the internet was fine...and i start looking and tell him jeezes man.... that guy is lost... im telling you...you know what ill go in there and you willl see its not the fiewall.... if we dont fix this he wont stop whinnning..
so i went in there...

i sit in a random computer and did ping 4.2.2.1... and there were slwo reply or anything...
i did Google and it took a while before it could resovle it.. so i say mm weird...
the net admin guide me to the owner office which was a pretty big one, and he tells me that the internet is slow so i said okay dont worry i ll fix it...

i did the same test on his computer 4.2.2.1 good!
Google super slwo then resolve it

then i did again Google and fast resolve it which was logic... it was on the cache...
i clear the cache and i do again... Google super slow
do again Google fast reply....
i do ipconfig /all
i see the internal dns server
so i changed it to the 4.2.2.1 and i test it again
Google fast reply
i browse everything good...
i tell the owner test it
and he was like wow now itsworking fast....

the admin ask me whats the problem..
i tell him the problem is your DNS Server
then he ask me how i fix it?
so i tell him open the dns server plz...
he log in the windowse rver and the dns server
i checked and instead or using root hints it was pointing to a random ip address inside his network... i ask him whats that ip?? and he tell me Let me check... he open a excel file and tells me id ont know whats that ip O_o i get rid of it and use the root hints... and then everything was workgin fine....
i ask him to test and everything is fine now...
it seems the dns server was trying to ask the random ip what was for example www.google.com and since it had no reply for while then after a while he used the root hints and thats why i guess it took a while.
anyways
and i ask the owner... since when you got this problem??? and he tells me well since many months ago...
and the admin was telling me that the problem was teh firewall O_o WTH
FAIL!!!
the sad think is that his is an international company which has LOT of branches around the americas.... i just cant bealive they get IT staff like this....
jeezes... all this was funny but also was frustrating... :P

like this i got many histories but oh well...

Anyone has an interesting history ????

varelg

None here yet, but don't be surprised by the staffing of certain companies cause that's what you get when you let people talk themselves into jobs.

Rearden

I see stuff like that all the time, either leftover from the previous admins who had the client before us or worse, when the client tries to do something in-house to save a few bucks.

gouki2005

any idea how this "administrators" get a job and they dont know what the heck the company network does or works

MentholMoose

A while ago a small business (200 users) hired my then employer, an IT consultancy, to replace the existing independent IT consultant who had been handling everything for them. For whatever reason he had the idea that the network in the main office (75 users) needed protection from power surges, so he started buying UPSs that would provide surge protection for a CAT5 cable, and started installing them. In some cases a device (PC, switch, etc.) was powered from it, but a lot of them were put in place JUST for the CAT5 surge protection.

Unfortunately he bought the worst UPS I've ever seen. It was a no name UPS on sale at Frys, and the CAT5 surge protection would corrupt or drop frames. This obviously started causing problems, and the solution was apparently to INSTALL MORE OF THEM. After installing dozens of these things, he couldn't figure out why the network was totally broken, and he got fired. The first thing I did after being hired was to remove all of those junk UPSs. All it took was a simple Ethereal capture to see how badly those UPSs were mangling the data.

azjag

A few years back I was tasked with removing some old servers from a data center. They had about 40 racks and needed to clear 5 out to make room for the new SAN. There were 21 servers, 3 NAS appliances, tape backup and kvms. We are talking high tech for the time, 36gb ultra scsi drives. So I proceeded to unrack all this equipment, fish out and remove the net cables from the floor, pdu's the whole setup. Moved everything into the designated corner to be "recycled". I gathered and sent the boss the spreadsheet of all the asseted items. It was pretty much an all day project.

Just before the leaving my supervisor came in to check on things, looked at the corner where I put all the recycled equipment and started laughing. Apparently I didn't need to do that much clean up. The next two hours were spent putting the racks back in place and re-installing the PDU's. They had started running network cables through ceiling mounted runs and had already run the fiber for these racks.

Made the install of the SAN much easier.