Department head is messing up weekly, whats our options?

--chris--

Quick summary, only facts that should help advise me/us (the IT department):

I started here about 3 months ago, very happy with offer/job duties and no red flags during interview. I was pretty picky about my next step as I had a good job prior to this. I still don't have any complaints with this place, very good people and job duties align with my long term goals.

My boss (VP of IT) started here 6 months ago. He comes from a similar institution where he was VP of IT for 12 years. This other place was smaller, about 1/3 the size we are. The vast majority of his experience is from managing the infrastructure of a much smaller organization.


Onto the problem:

Our VP has been ignoring our change procedure and making changes at the branches and in our DC to systems that have created company wide outages on a weekly basis for the past two months. Some of the issues have garnered attention from his peers and other high level people. We covered for him the first few times and basicly said, we are aware of the issue and are working to resolve asap....now that the issues are occurring frequently and we are getting calls weekly to fix what he is breaking its getting harder to cover for him because its making us look bad.

We have approached him and offered suggestions to enhance his approach to change (bring us into the plan early on, let us know what you are going to do, when you do it, etc...), offered to do the work ourselves, offered alternative solutions (I.E....maybe its not a great idea to have every branch point to a different DFS server...because collisions you know?)....but none of these suggestions have been considered. He thinks we are afraid of change and unwilling to "break a few things to make it better around here".

Honestly, we have no confidence in his ability to lead us or effectively maintain the IT department.

Now that we have approached him and been rejected a few times, we feel we should escalate this...but how? He has a ego problem, bullies subordinates and is stubborn. I can deal with all of that, no problem...I have dealt with much worse. However, my peers do not feel the same way and are actively looking for other jobs (one is going for a second interview this week). If they leave, my job will definitely get harder and the cycle will continue when they hire new people.

The options (I am open to other ideas!):

1. Go to his boss (COO) and ask to speak in confidence, lay out facts without emotion and quantify the worst possible outcome with each change he is making (or not making).
2. Go to HR and do the same as above (I said no way to this, HR is not here for employees).
3. Sit down with our boss and lay it all out, hope it hits him and he "sees the light".


Obviously we all fear for our jobs, none of us feel irreplaceable and would prefer to do nothing if we knew it would mean losing our jobs.

Any suggestions or past experience that might help me out?

Examples of things that have happened (not a complete list, just some I remember):

• Added servers to the DFS replication (branch servers), added large directories to DFS; this caused branch connectivity issues because the DFS replication was using 100% of the available bandwidth (2-3 Mbps). Phones were up/down, connection to the "core" software was constantly broken, a few branches reverted to offline processing. This was 2 days in length. None of us new this change was made until he mentioned it might be causing an issue.
• Changed phones to DHCP at a branch for testing; was then unable to successfully configure DHCP on the voip box. This broke phones at a branch for 4 hours while someone drove out there to change them back.
• Changed a different branches workstations to DHCP w/mac addy reservations but did not let anyone in the department know this; branch lost power and the server never came back up; us assuming the workstations were still static like they have been for a decade spent 45 minutes troubleshooting other possibilities; total outage was 2 hours
• Added a new (empty) folder to DFS that overwrote a key folder, replicating this change out to all DFS servers; this broke a vital piece of software for 5 hours
• Changed the gateway IP address on a branch server to the wrong gateway at 3 PM; branch stayed up because of how DNS & profiles are setup but someone again had to drive out there and fix it.
• Moved 30 users profiles to a group of DFS servers that do not participate in any backup; when I suggested to him we include a VM file server in this group so that can get Veam to back that up he said "DFS is a backup" and "why not put a NAS at a branch and robo copy the directory over once a week?" along with "we tried VM file servers at the last place, we had all sorts of issues...they wont work for us".

scaredoftests

Wow, that is a toughie. First go to YOUR boss and talk to him (and document it). Hopefully, he will move it up the line. If not, go to HR.

JoJoCal19

I'd say option 1. And I'd make sure you have your request to speak to COO documented via email so you have a paper trail in case there is retaliation. Reason I say skip option 2 is because I agree, and with option 3 you could immediately piss him off and he could make your work life hell (based on what you said). At least with option 2 the problems will be well known and documented at a higher level. If you go straight to him he could just can you immediately and upper management would be none the wiser.

--chris--

scaredoftests wrote: »

Wow, that is a toughie. First go to YOUR boss and talk to him (and document it). Hopefully, he will move it up the line. If not, go to HR.

I neglected to clarify, the VP of IT (department head in question) is my boss. So going to my boss would be option 3 above.

IronmanX

Tough one.
You go over his head and even if his boss is discrete you and your team is going to have to deal with a boss who knows people on his team went over his head. I hope he has no say on performance reviews and raises.

Don't go over his head and the organization will think you guys are doing a bad job.

--chris--

IronmanX wrote: »

Tough one.
You go over his head and even if his boss is discrete you and your team is going to have to deal with a boss who knows people on his team went over his head. I hope he has no say on performance reviews and raises.

Don't go over his head and the organization will think you guys are doing a bad job.

He does the reviews and raises. It does NO ONE any good to have an adversarial relationship with him or him with us. Finding the right way to approach this is tricky.

The more I discuss this with people the more I think this is just a part of the job. Like I said, I can deal with it...I am getting really good at troubleshooting because of it.

IronmanX

Sucks that you might lose good people over this.
When someone leaves get them to talk to upper management about the reason they are leaving and leave everyone else out of it haha.

scaredoftests

Holy cow. That is a tough one. You can go to HR, but that can backfire..

DDStime

All in all never go around the chain of command. I have seen people blackballed or fired over this.

Be an adult and sit down with him and voice your concerns. See what happens in a one on one with him. It will take a lot of courage, but you might get the situation resolved.

Going around him will make him look like an idiot (which he is) but will also make you look like an idiot. If someone went around me and told my boss bad things about me without addressing the issue with me first I would fire them immediately.

Or you can just use the opportunity to learn how to fix the huge issues he is causing....kinda like a home lab lol

beads

@--Chris--

First document everything, every change, every fault. Clearly these "little problems" are noticeable to everyone in the company so no worries there. Allow whatever business system in place to work its way through itself. Raising a fuss at this point will only make you look like a mutiny - which in this case - it is. Hold off and be patient.

Every time one of these obstacles occurs simply start the business case for the remediation. Not enough bandwidth to remote locations? Really? Put in the request and justification for more bandwidth and another person to manage it if needbe. Nothing like increasing expenses in an organization to gain visibility.

Broke DHCP phones. Too easy. Your describing poor design with no remote support, ie LogMeIn, RDP server. Sorry, can't help bad or unresistant design practices. Again, push through the corrected design and take credit for saving money in the future. Lost productivity is painfully expensive to a business. Be a hero and not pizza delivery having to speed to a downed off site to flip a switch or two. Hero status when you put into dollars and cents or sense. Your pick.

The last three beg to ask why you do not have a Change Management System or CAB or whatever. You have a VP of IT making tactical decisions best directed not enabled - these are just plain silly. Not to mention show a very low CMM. Push through a CAB system or similar so that these types of changes are no longer performed under the darkness of night.

Downside. If not done to your betterment you will likely make your boss/putz look like a management guru only adding gasoline to a blazing fire. You and your group need to be seen as championing these changes otherwise your enabling an already poorly performing manager and it sounds as though this person is really your manager with an inflated title. A VP usually has 2-3 Directors reporting to them not analysts.

Best way to do this is to briefly forewarn the boss shortly before a bigger meeting with him/her, the COO and as many other management team members in the same room as possible to you have witnesses to your little endeavor. Yeah it takes some brass to pull off but like any other vengeance, its a dish best served cold and garnished aplomb.

Think smarter than the technology you are running and you will win. Otherwise be prepared for the next fiasco.

Been there dumped that manager.

- b/eads

brewboy

Afraid there's not much you can do. And I agree, never go to HR for something like this. Actually, I'm surprised a VP is making changes like this.

IronmanX

DDStime wrote: »

All in all never go around the chain of command. I have seen people blackballed or fired over this.

Plus going above him realize that guy above him hired him. Saying hes screwing up and is not good etc... is probably going to put the COO on the defensive and make you look bad in his eyes.

networker050184

Start polishing up the resume and putting out some feelers first and foremost. After that I'd go with approach #1.

the_Grinch

I agree that this isn't an HR issue. I would suggest going directly to your boss and respect the chain of command. Two sides to every story and I would think any one (regardless of their position) would want the benefit of the doubt. Document that you had the meeting and see if the issue is then resolved. If not then I would go ahead and approach the COO with the issues. In the event that there is retaliation you'll have clear documentation showing that issues did not start until after the meetings in question.

Often we do not know why a manager decided to do something. I've been in situations where a boss made a change or did something solely because those above him said "I want x or y now". But I agree brush up the old resume just in case because if nothing else I wouldn't want to stay a company where policies and procedures aren't followed.

gespenstern

The last one, sit, witness, document and wait until he gets fired. If not -- then the company is okay with his approach and who are you to deny them from doing that? Do you own the company? No, you are salaried and is expected to do what you are told to. If they decide to do it the hard way -- it's their company and their choice -- politely express you opinion if asked (which you already did) and let them do what they are up to.

You can be 100% right and 100% fired. Don't be, just do your job and let them run their business how they want. If they don't do it right they will eventually go bankrupt and the market will kill them. If they won't then they were actually right and you weren't.

MeanDrunkR2D2

Personally, I'd not bring it up and let those fires continue to pop up here and there. He's your boss and you should happily do what he requests of you. Gain his trust and he may start to bring you into those decisions when he sees that you are the one that is capable of fixing those issues. Sure, you could jump around to his boss but all that may end up doing is causing him to not like you and make your life hell. HR is a no go because they are never on your side. Talking to him, well, it sounds like you already did so. Just wait until the next time he blows something up and try to sit side by side with him to troubleshoot the issue. Have him trust and respect you and you'll soon be the one he may confide in on those decisions.

And, it probably could be a case where he does know what he's doing, he just has a completely different technique to quickly get things fixed. And it could be that he's trying to show these "issues" to those with control of the financial wallet to open it up to make necessary changes to improve things there. Sometimes a manager/VP will strategically "break" something that may cause minor headaches, but will get those higher up the food chain to do something about it.

bigdogz

I would brush up the resume and just keep working issues.

fredrikjj

Before going with option 1 you need to figure out how the COO feels about this problematic VP. The COO was perhaps skeptical of this hire in the first place, but was overruled by the CEO (maybe the VP is the CEO's old college buddy). You going to the COO with this stuff is perhaps just what the COO is looking for to get rid of the VP. Or, maybe it's the other way around and the VP is the COO's old college buddy, and the COO has invested a lot of prestige in bringing him on and the CEO was the skeptical one, making it impossible for the COO to fire him without losing face. There are so many variables.

sj4088

gespenstern wrote: »

The last one, sit, witness, document and wait until he gets fired. If not -- then the company is okay with his approach and who are you to deny them from doing that? Do you own the company? No, you are salaried and is expected to do what you are told to. If they decide to do it the hard way -- it's their company and their choice -- politely express you opinion if asked (which you already did) and let them do what they are up to.

You can be 100% right and 100% fired. Don't be, just do your job and let them run their business how they want. If they don't do it right they will eventually go bankrupt and the market will kill them. If they won't then they were actually right and you weren't.

This sound good in theory. But the problem with this approach is all the people under that VP is going to be called on to do a lot more work to clean up his mess. So I can see why this is a problem for him and the team.

Queue

Going above your bosses head is insubordination and you can't walk back from that. Management tends to stick together from stories I've heard, and experienced. I would attempt to make it work with the boss. I read another comment in the thread I liked best. The gist was along the line as try to prove to the boss you have decision making ability and maybe you will be his go to guy. I don't know your experience, but neither of you have worked for the company very long at all.

networker050184

sj4088 wrote: »

This sound good in theory. But the problem with this approach is all the people under that VP is going to be called on to do a lot more work to clean up his mess. So I can see why this is a problem for him and the team.

Not only that when it finally comes to the point where heads have to roll for a big mistake you think he's going to fire himself?

blargoe

That is a horrible situation to be in. If it were me I would be working on moving to a different company yesterday.

When he gets called to the carpet be senior management about all the recent infrastructure issues, I doubt he is going to fall on his own sword. Working for an combative, incompetent buffoon can be miserable, but him jumping in and literally screwing things up without regard to the impact on the business is something I can't wrap my head around. I'm not one to just give up easily but I don't know that are likely to win this one if you stay.

OctalDump

I'm from a different work culture, so I am surprised a bit by this situation and the recommendations.

I'd expect that the workplace would have in place structures/policy/process for dealing with these situations. Bosses do mess up, they sometimes do criminal things (embezzlement, sexual harassment, fraud, bullying etc), so a workplace needs support in place for employees to talk about possible bad things bosses do. A mature organisation would even have whistleblower processes in place.

It sounds like he is breaking company policy by ignoring the change process - the change process should have ways to do deal with this ie reporting and sanctions. A policy without teeth is merely suggestions.

The other thing I find odd, is that the kind of "obey all orders, no questions" attitude. I'd hope for a more collaborative approach, where it is easy for someone to disagree with their boss - even if the boss has final say. Possibly the ease, and willingness, to fire people for any minor infraction or upset is part of the problem. It's crazy to me that an organisation would lose someone with real value, to hire someone else that would take months or years to get up to speed.

All of this sounds like an organisation willingly engaging in practices that are against its own interests.

What would I do in a situation like this? Form a united front with my co-workers and have a meeting (more like an intervention) with my boss. It might be easy to dismiss the views of one employee, but harder when there's 3 or 5. It also makes it easier to have the conversation by shifting the power dynamic. But you want the meeting to be focussed on 'solving the problem' rather than berating the manager. I'd also explore what options are actually available for reporting bad behaviour by a manager. But like I said, I'm from a different work culture, so I'm not sure how that would play out in reality for you.

UnixGuy

Are you still learning and improving your skills? It seems like you are, and you're learning valuable troubleshooting skills.

Why are you upset?

tmtex

Keep your mouth shut, do your job. Eventually ..it will hit the fan. Keep emails etc., Document for when that day comes

IronmanX

OctalDump wrote: »

It sounds like he is breaking company policy by ignoring the change process - the change process should have ways to do deal with this ie reporting and sanctions. A policy without teeth is merely suggestions.

The other thing I find odd, is that the kind of "obey all orders, no questions" attitude. I'd hope for a more collaborative approach, where it is easy for someone to disagree with their boss - even if the boss has final say. Possibly the ease, and willingness, to fire people for any minor infraction or upset is part of the problem. It's crazy to me that an organisation would lose someone with real value, to hire someone else that would take months or years to get up to speed.

All of this sounds like an organisation willingly engaging in practices that are against its own interests.

I'd like to hear how people have dealt with this same issue. Sounds like most people are just saying move on to other job there is no one to resolve the issue.

I for one see no way of reporting the issue to higher management. That management picked him to do this job. The bad thing is when he gets called out by upper management to discuss whats going on the people under him are not going to be part of that conversation.

The only way to resolve this problem is do like what someone else said gain the bosses trust and suggest a CAB or something. But if he says no or says yes and then decides not to follow it you really have no recourse.

Of course if the boss is doing something illegal then you should be going up to HR with proof, but this is really a ummm management style problem haha.

--chris--

First, wow! The responses and thouroughness in explaination are appreciated. Thanks everyone! I can't respond to all the comments, but I have read them all and have factored them into this equation. I will try to respond to the replies that will answer questions and help others offer advice to me.

beads wrote: »

@--Chris--

Advice....

- b/eads

We have started to document all failed changes, root causes, resolution, total down time, etc...I manage them for the team.

I really like your approach to this problem. Essentially: trying to make things work no matter what and documenting what we do to that end, right?

We are in finance (PCI compliance is required for us). Up until I got here they did not have a change management policy. My second week here I drafted one and discovered they were dinged for that last year and had it as a "project" to complete before this years audit (4 weeks away). I just got it implemented last week, but as another posted said it has no teeth therefor he is not following it.

Can you expand on this a little? I don't understand.

Downside. If not done to your betterment you will likely make your boss/putz look like a management guru only adding gasoline to a blazing fire.

--chris--

IronmanX wrote: »

Plus going above him realize that guy above him hired him. Saying hes screwing up and is not good etc... is probably going to put the COO on the defensive and make you look bad in his eyes.

This was brought up during a strategy meeting we had about this problem. We could probably go to the COO without issue, the CEO however choose him and would most likely not take it well.

--chris--

networker050184 wrote: »

Not only that when it finally comes to the point where heads have to roll for a big mistake you think he's going to fire himself?

This is another real concern we have, sometimes it feels like a "damned if we do, damned if we don't" thing.

--chris--

UnixGuy wrote: »

Are you still learning and improving your skills? It seems like you are, and you're learning valuable troubleshooting skills.

Why are you upset?

I am not that hung up on the lack of change control of the constant fire fighting I do. What will make me upset is losing two co-workers and having to train up two replacements that will take 3+ months when I only just started here. That won't be fun.

For the record, another change occurred last night which caused a branch wide issue. When we asked the VP what changed he did not answer and just asked who was affected then said he will take care of it.

MeanDrunkR2D2

You're in a hard spot. I don't see a good way to bring this up to him without him taking it the wrong way. Just do as he asks of you and just CYA in case he breaks something major and you can point to a paper trail that shows who did it if he tries to throw you under the bus. If he tries to throw someone else under it, so be it, it's not you. Keep learning, gaining knowledge and you could have a very good person that could help you out more in the future should he go elsewhere and he knows that you will fix anything and do it right.

Too many always think that if things get hard, or they have a boss who has a different approach that they should just leave. If you honestly like your job, and as a person you like your boss and the company why sabotage that? He just may have a different approach that has worked for him in the past. I work in finance as well, but luckily my department has a very strict CM process so we don't get dinged for PCI issues in that regard. Yes, things break, sometimes badly, but you need a damn good team that can follow direction and fix the issues. I've seen horrible infrastructure in place at some companies and the only way to really get the changes needed and to loosen the purse strings of those in control was to have things break that needed to be upgraded. His strategy could be very purposeful in the long term. You and your co-workers may just be seeing the short term issues while he's looking a year in the future and where he wants the department.