Company Burning You with Procedure?

the_Grinch

Allow me to describe what happened and the interaction there after. First, there was a screw up on my part so I am not without fault on this one, but believe the company is coming down on me for procedures that aren't documented and change often (or have so many exceptions you don't know what to do).

Part of my nightly duties is to check our customers backups. I check probably about 75 backups every night for completion and make tickets for the ones that failed. Today a customer emailed in about not being notified that her backup had not completed since Friday. My manager said he was going to review and then tried to slam me in an email for the problem. When I took this position the following procedure was in place:

Check each backup and update backup sheet (there are two as we monitor two types of backups)
If the backup fails, mark it on the backup sheet, mark it on the daily hand off (is an email with a list of issues that occurred during the night)
Create a ticket for the failed backup

Now the guy I took the position from had a couple bad habits so the procedure had some amendments made. First, under normal circumstances when we make tickets we email the customer from the ticket. Due to the former night guy not truly checking the backups, he'd tell a customer the backup failed, when it hadn't. So the procedure was for the person working on the ticket to review the issue, confirm the failure, and then contact the customer to let them know. Second, he also had a problem assigning the wrong priority to the tickets and due to when the morning shift comes in we'd miss our SLAs. So, he would assign the ticket to the morning person who comes in at 6 AM (on Wednesdays, we have two people in at 6 AM). This allows us to not miss the SLA and keeps the board clear of the tickets since two techs basically work on backups fulltime.

On Friday (I work Monday thru Thursday so I don't check Friday backups) the night NOC guy didn't create a ticket, put in the report, or in the hand off that the backup failed for the customer. Now he's been consistently not performing the backup checks and has been caught sleeping on the job. Now where I screwed up was on Monday I checked the backups and noted in the report that the customer's backup had failed. The issue is I didn't put it in the hand off and didn't make the ticket (no idea why, definitely my screw up that I owned up too). On Tuesday their backup failed, I placed it in the report, placed it in the hand off, and created the ticket. I then assigned the ticket to one of the two techs who is in on Wednesday (as I have for three months now). Last night was terrible (if you live in the NJ, Philadelphia, NYC area and deal with Paetec their circuits were bouncing for hours) and I was dealing with 1200 alerts. Once everything settled down, I had 5 customers with outages that had to be called out. Now, I had completed my backups and assigned the tickets (out of 75 backups, we had 4 failures). One of the techs didn't show this morning (he didn't call so I didn't know he wouldn't be there) so the tickets I assigned to him stayed on his board. I stayed an extra hour to get the circuits called out then headed home.

I wake up and I get an email from my manager in regards to the customer stating we didn't notify her that the backup had failed. He stated he had to search for the ticket and saw that I assigned it to the tech that didn't show up. He then stated that I am suppose to notify the customer that the backup failed and never assign a ticket to someone unless they ask. Now I'm pissed, I realize I failed on Monday, but this issue started Friday. Thankfully I took a run prior to responding because my reply would not have been a great one. When I did reply, I stated the following:

1. I had been told that those working on the backup would notify the customer
2. No one had told me that the tech wasn't showing up (we are suppose to call the NOC and our Manager to let them know we aren't coming in), had I been told I would have moved the tickets off his board
3. I noted the failed backups in the hand off report (which I've said that the minute we let everyone have it, they'd start ignoring the report) so it should have been caught
4. I then noted 5 tickets of failed backups where the night NOC person did not email the customer about the failure (all prior to my taking the position)
5. Finally, I stated I was following the procedure as laid out to me and that I didn't have a problem with them changing it. I also stated that it wasn't like I came up with doing things this way

Am I wrong here? I mean I definitely didn't follow the procedure on Monday, but this failure didn't rest solely with me though they appear to be trying to make it out as if the total failure was on my part which it was not. These things seem to occur often here and it makes me not want to stay anymore. I've requested more help at night and have been ignored while they talk about getting the weekend person help (I just started the job and haven't fallen asleep once, also haven't missed acknowledging an alert in three months. Though I did miss making that ticket, which I can't for the life of me figure out why!).

What I decided to do now is to make a ticket for every alert that we get. Basically, no procedures are written for how to handle the various alerts and for the most part I would clear them without flooding the board with tickets for the day shift. But since they appear to be trying to burn me, I plan on following the standard ticket procedure and not go beyond what is required. Getting tired of the different standards set and expected of coworkers....

KrisA

I have seen the day verse night, 1st shift verse 2nd verse 3rd in pretty much every sector I have worked in. I think the majority of the time people want to "out perform" their counter parts.

All you can do is not dwell on it. Ask for official policy/procedures or find the SOP (Standard Operating Procedure) and do your best. I have assigned tickets to individuals and been burned when they are out of office. So I changed to assigning to the group ( if you have that option) so the entire queue/group then sees the ticket and can take the appropriate action.

not go beyond what is required

I wouldn't recommend this as you are then allowing them (whom ever it may be) the control the situation. Then the complaint will come down that your only doing the minimal amount of work to get by.. ( Office Space and pieces of Flare comes to mind). Continue to be the person that gets things done and gets them done right. It could also be a situation where you have not had any faults they could capitalize on, and the opportunity to bust your chops came up so they did it. Don't get into, for the lack of a better expression, **** match. It is not worth the time, energy and extra resources you have to put into it. You made a slight mistake, you owned up to the mistake and your now more aware of what your doing in regards to the alerts so future issues wont happen.

the_Grinch

Solid advice and you're right I shouldn't go about doing the minimum. My main issue is I know what was done/expected of my predecessor and to see a different standard applied erks me a bit. I also see what gets missed during the day and I have to assume it isn't being addressed as it continues to happen. I figure at this point I could probably dwell on all of that and stew in it or get over it and worry about myself (which seems to be the best course of action). Thanks for the advice!

knownhero

Sounds like the place I currently work at. Changing everything without telling anyone and wonder why you messed up when you have been doing the same thing for 4 years. Id take blame for what I did wrong and then pass of the rest as failure of a group of people and this doesnt rest just on you.

Dwelling on these sorts of things will drive you crazy! And really not worth it.

Devilry

knownhero wrote: »

Dwelling on these sorts of things will drive you crazy! And really not worth it.

I agree, this will drive you mad. Brush it off, as much as you can.