Company Burning You with Procedure?
Allow me to describe what happened and the interaction there after. First, there was a screw up on my part so I am not without fault on this one, but believe the company is coming down on me for procedures that aren't documented and change often (or have so many exceptions you don't know what to do).
Part of my nightly duties is to check our customers backups. I check probably about 75 backups every night for completion and make tickets for the ones that failed. Today a customer emailed in about not being notified that her backup had not completed since Friday. My manager said he was going to review and then tried to slam me in an email for the problem. When I took this position the following procedure was in place:
Check each backup and update backup sheet (there are two as we monitor two types of backups)
If the backup fails, mark it on the backup sheet, mark it on the daily hand off (is an email with a list of issues that occurred during the night)
Create a ticket for the failed backup
Now the guy I took the position from had a couple bad habits so the procedure had some amendments made. First, under normal circumstances when we make tickets we email the customer from the ticket. Due to the former night guy not truly checking the backups, he'd tell a customer the backup failed, when it hadn't. So the procedure was for the person working on the ticket to review the issue, confirm the failure, and then contact the customer to let them know. Second, he also had a problem assigning the wrong priority to the tickets and due to when the morning shift comes in we'd miss our SLAs. So, he would assign the ticket to the morning person who comes in at 6 AM (on Wednesdays, we have two people in at 6 AM). This allows us to not miss the SLA and keeps the board clear of the tickets since two techs basically work on backups fulltime.
On Friday (I work Monday thru Thursday so I don't check Friday backups) the night NOC guy didn't create a ticket, put in the report, or in the hand off that the backup failed for the customer. Now he's been consistently not performing the backup checks and has been caught sleeping on the job. Now where I screwed up was on Monday I checked the backups and noted in the report that the customer's backup had failed. The issue is I didn't put it in the hand off and didn't make the ticket (no idea why, definitely my screw up that I owned up too). On Tuesday their backup failed, I placed it in the report, placed it in the hand off, and created the ticket. I then assigned the ticket to one of the two techs who is in on Wednesday (as I have for three months now). Last night was terrible (if you live in the NJ, Philadelphia, NYC area and deal with Paetec their circuits were bouncing for hours) and I was dealing with 1200 alerts. Once everything settled down, I had 5 customers with outages that had to be called out. Now, I had completed my backups and assigned the tickets (out of 75 backups, we had 4 failures). One of the techs didn't show this morning (he didn't call so I didn't know he wouldn't be there) so the tickets I assigned to him stayed on his board. I stayed an extra hour to get the circuits called out then headed home.
I wake up and I get an email from my manager in regards to the customer stating we didn't notify her that the backup had failed. He stated he had to search for the ticket and saw that I assigned it to the tech that didn't show up. He then stated that I am suppose to notify the customer that the backup failed and never assign a ticket to someone unless they ask. Now I'm pissed, I realize I failed on Monday, but this issue started Friday. Thankfully I took a run prior to responding because my reply would not have been a great one. When I did reply, I stated the following:
1. I had been told that those working on the backup would notify the customer
2. No one had told me that the tech wasn't showing up (we are suppose to call the NOC and our Manager to let them know we aren't coming in), had I been told I would have moved the tickets off his board
3. I noted the failed backups in the hand off report (which I've said that the minute we let everyone have it, they'd start ignoring the report) so it should have been caught
4. I then noted 5 tickets of failed backups where the night NOC person did not email the customer about the failure (all prior to my taking the position)
5. Finally, I stated I was following the procedure as laid out to me and that I didn't have a problem with them changing it. I also stated that it wasn't like I came up with doing things this way
Am I wrong here? I mean I definitely didn't follow the procedure on Monday, but this failure didn't rest solely with me though they appear to be trying to make it out as if the total failure was on my part which it was not. These things seem to occur often here and it makes me not want to stay anymore. I've requested more help at night and have been ignored while they talk about getting the weekend person help (I just started the job and haven't fallen asleep once, also haven't missed acknowledging an alert in three months. Though I did miss making that ticket, which I can't for the life of me figure out why!).
What I decided to do now is to make a ticket for every alert that we get. Basically, no procedures are written for how to handle the various alerts and for the most part I would clear them without flooding the board with tickets for the day shift. But since they appear to be trying to burn me, I plan on following the standard ticket procedure and not go beyond what is required. Getting tired of the different standards set and expected of coworkers....
