Change Management -- Your experiences good and bad

Geek1969

What are your experiences with change management?

The company where I work has a formal change policy and procedure with a change advisory meeting twice weekly. Submit your request for change(RFC), appear at the meeting, explain your change, have it approved or denied, implement or revise your RFC.
I believe that this process is needed and should be strictly adhered to whenever possible (emergencies withstanding).

We always seem to have numerous undocumented changes happening without going through this process. Network config changes, software pushes, code releases, devices added or removed from the network ..etc. Nearly every week we find something that was changed or added (sometimes causing outages) without an approved RFC. There never seems to be any consequences for causing outages, with or without an RFC.

What does your company do in regard to Change Management? Do you have unauthorized changes happening? What are the consequences if there are unauthorized changes?

tpatt100

We have a policy in place for some changes but not all. It is part of what I am working on currently to improve/fix but meeting resistance to change.

m3zilla

Meh - that's how I feel about it.

At my company, we have a department dedicated to change management. Every change goes through a workflow where it needs to be approved by multiple people before we can implement it during our maintenance window. Where I work, rarely, will you be making a change without an approval from change management because if something goes wrong, and you don't have an approval for your change, you are in deeeeeep trouble.

I get the idea behind change management and the need to track changes to the network, but at times, it feels like it's just there to make my life harder.

J_86

My "change management" is; "If I make this change I better know what is going to happen (tested in a lab, if possible) or it's my a$$ if I take down the network". It's mainly just me at my work, so if something breaks guess who's fault it is?

On a seriously note.

I think it also depends on what kind of environment you are working with. If you are working in a large enterprise type environment, change management is vital. The more hands you have changing things the more that could go wrong and you need to know where to start looking if something does go wrong. If you are in a small environment change management is still important so you are keeping track of your network and documenting it, but a lot of things can slide.

ChooseLife

Having a functional change management process is the only way to survive in an environment with multiple moving parts and multiple entities driving and/or implementing changes.

The problem of unauthorized changes is not an easy one to solve... Generally, the management needs to be interested in addressing the problem. If the management does not feel there is an issue or actually supports such behaviour ("problems need to be fixed asap", "we can't wait 3 days for a simple change to get done"), not much can be done.

Now, if the management supports the idea of getting change management done properly, there are a few options:

a) enforcement through disciplinary actions
b) separation of duties - entity requesting a change cannot carry it out due to the lack of technical access, another entity implements the change once it's approved by CAB (this is commonly implemented as Development vs Release team, Design/Engineering vs Operations team)
c) control of the ability to make the change - the entity requesting a change also implements it, but a technical ability to perform the change is given to the entity by another party after CAB approval (temporary access to production is granted by operations team to an engineering team)

HTH

jibbajabba

Being in a big ITIL fanatic corporate, implementing changes without prior CAB approval can have severe consequences to the business and yourself as individual. There are 'Standard' changes which can be approved by our direct manager, changes which are known not to cause any impact to the business. Although even there is a pre-defined list. Emergency changes still require CAB. It is different with 'break fixes' - i.e. at 3am, systems are down because of xyz. If you got a fix right there then it can implemented. But that usually doesn't effect the business as a whole and when in doubt we do have to escalate, even if we know we can fix it.

lordy

We have a pretty simple procedure without too much paperwork. Every Change, that is not Access Rights (like creating, deleting or modifying user accounts) is put into the next weeks change plan by the person that requests it. It is then sent to their Manager and the the Requestor briefs the Manager on the details, if necessary.

Every Friday afternoon the Managers come together as the CAB and discuss the changes that are in the plan for next week and decide wether they can and should be implemented.

We still have some Changes that don't go through the process because people consider them trivial or there are some even higher-ups screaming that they can't wait until next week to have this implemented. You probably know the drill...

DevilWAH

The level of change management must be correct for the comapny involved.

For very large "global" systems, where a change on one site can have major effects on other sites, and in cases where an outage could cause major loss of buisness/money. Then effective change managemnt is critical. This may need to involve mutiple steps and mutiply aprovers.

However for other buisness then a simple excel spread sheet may be plenty.

In the same way that every company no matter its size or role should carry out a security exercise and develope its security policy, the same is true of change management.

In an IT unit of 3 people, you are not going to need a system that communicated changes around the team so much, here a spread sheet and email + meetings my be fine.

Make a change mangment sysstem to complex for whats needed and people will try to bypass it, ignore change managemnt and you could end up in a mess later.

All change management is, is planing and recording cchanges. there are no hard and fast rules for how indivual compnies go about it.

Good change mangemennt systems also make allowance for common changes, lowing them to bypass some or all of the aproval process and jsut get recorded.

the_Grinch

I can honestly say that I have never seen change management that seemed to work. But I usually equate it to a couple of things. The biggest thing is the platform you use to record you're changes. If it's isn't easy and convenient people won't do it. Lazy technicians would be my other gripe. At my last job (MSP) things were stored in several different places and people were just too lazy to update the info. Getting needed notes in tickets and time sheets updated was difficult. Finally, band aid fixes tend to blow change management out of the water. Tech gets a call at 2 AM to fix an issue, fixes it, goes back to bad, and for months it works. All of a sudden everything goes down and you have no idea why the change was made and why a permanent fix was in place.

What I would really like to see is a system that would detect changes made to various devices and then log it. From there you would get an email stating this change was made and ask for documentation to be completed. If not completed in x number of days, another email gets sent and management is notified. Probably not easy to make, but boy would it be nice.

DevilWAH

the_Grinch wrote: »

What I would really like to see is a system that would detect changes made to various devices and then log it.

I have worked on systems where ever command you enter is logged centraly, any changes in configuration of device is sutomatily noted and sent to the change mangment team where it is (automaticly) cross refrenced with the change mangment ssystem to see if it is an authorised change.

Any one making cahnges with out the correct authorisation gets a strike and its a three strike and out policy. Any one getting 3 strikes has there accounts suspened untill they have redone the Change mangment system and policy training.

all servers and network devices can log who has made changes easly, its then a simple matter of pulling it back to a centrlise location, either via SNMP or syslog, or in the case of cisco using AA to centrlise authorise and account for every command entered on the network. (sure windows has some thing simmler)

DevilWAH

While we are talking about it does any one know any good / simple change mangment systems? are then any good opensourced ones around?

I ahve only worked with in house built systems or Remedy systems costing millions. Looking to implement one at work and jsut would like a few to test out that wont break the bank.

Tackle

What is this Change Management you speak of? Haha. Must be a larger company type of thing.

If something is removed, added or changed, I do it, make a note if needed and am on my way.

blargoe

Most anything I do requires a change request. I can't plug a server in without a change request.

dave330i

blargoe wrote: »

Most anything I do requires a change request. I can't plug a server in without a change request.

DRS & sDRS set to manual?

N2IT

Most mature organizations have some sort of change management process in place.

I lean towards the side of having a strict policy rather than a laissez faire one in place. There should be defined regular request and then of course you have you release management piece that should be documented for patching etc.


All changes to the infrastructure of any significance should be track IMO

ptilsen

Agree with N2 here. The policy can be designed to accommodate urgent requests and low-impact requests, but the implementation should be strict and well tracked. It needs to be a part of/tie directly into the CMDB so that changes can associated with the pertinent assets and assets with the pertinent changes.

blargoe

dave330i wrote: »

DRS & sDRS set to manual?

Why, yes, it is. But I think I've actually convinced people that vMotion is not a bad thing though, so I might be able to turn DRS back on (conservative) once I'm able to get all of these damn VM memory reservations turned off...

CodeBlox

Change management is a huge hassle... Had to deal with this at my old job. New job doesn't rely on it so much and I like it a lot more. Change management at my old job just seemed like a way to waste everyone's time and get some lucky person a job.

jibbajabba

A lot of companies now adapt service-now.com - including us.

atorven

I get the whole change management process for big changes but how about the small changes? How about when you are troubleshooting a mission critical issue, do you go through the whole process for every single step you need to try or do you just keep track of the changes you have made and log it at the end?

ipSpace

CodeBlox wrote: »

Change management is a huge hassle... Had to deal with this at my old job. New job doesn't rely on it so much and I like it a lot more. Change management at my old job just seemed like a way to waste everyone's time and get some lucky person a job.

Well it depends on the company. In my opinion it is perfect, this Change Management things..

I work for a multinational company, and it is really hard to follow on what other people (from different countries) are doing without the Change Management stuff.

DevilWAH

atorven wrote: »

I get the whole change management process for big changes but how about the small changes? How about when you are troubleshooting a mission critical issue, do you go through the whole process for every single step you need to try or do you just keep track of the changes you have made and log it at the end?

You are talking about indents. this falls out side change management.

Generally change management looks after planed changes and updates.

So for indents you just get on and fix the issue and don't worry about the change process, its a case of get it back up and working. Once it is working if you have made any config changes to the devices (that is different to before you started, not every little thing you have tried) , you would then add this to change management system and it would be approved retrospectively.

So it still have a record of any actuly changes to the systems, some incident management help-desk feed directly in to the change managment database, so for example a incident ticket is opened. You pick it up,resolve it, complete the tickets and this is then stored as a recourd.

In terms of small changes you generally set up change templates that are pre approved, and/or may be coded in to the helpdesk system so are added automatically. So take the example of enabling and patching a port for a new user.

With in the help desk ticket there may be some fields you need to complete, socket number, switch port, department... Once you have completed the case then this information is populated in to the change management system.

Or directly in the change managment system you will have set up some changes that have been pre-approved, maybe adding a new PC to the domain. So originally one person will create a change request, and ask all the approvers to pre approve all further changes along with a list of people trained to carry it out. few different names for these.

Fast track changes
Standard model change (SMC)
Standard Request (SR)

And as I mentioned may companies use the help-desk systems to gather data for these types of changes, so you might be doing change mangment with out ever knowing it.

Geek1969

Some great insights here people. Thanks!! Most of you that have posted seem to operate in some form of change documentation environment depending on the size of the company. I understand the people who say CM is a pain in the $#@&. I agree to a point. More importantly though, CM can cover your $#%& if the procedures are followed. To answer a few people, any "planned" change "that will affect more than a few people needs to be pre-approved. (2 people losing network access is not an outage...no RFC). Rebooting the dhcp/dns server needs a pre-approved RFC. Any "unplanned" changes due to necessity or outage (hardware failure, power outage, network attack, human error, etc) need to be documented in a "post-mortem" RFC after the changes were implemented to restore service. The specific problem I see often is an unplanned change without an approved RFC being implemented to fix a minor inconsistency, that accidentally causes a large network outage. Any similar experiences? How does your company handle this?

DevilWAH

Any unplanned or planned change carried out with out a correctly approved RFC, will result in a mark against the person at fault. May be the person carrying out the change going ahead before approval, or not post an unplanned change documenting it. Or against an approver approving incorrectly aproving a change.

In each case if it is a simple process error then its 3 strikes before you get locked out of the system and have to redo the training.

IF you unplanned / recorded change causes and outage.... Well then it depend how bad the outage is and how kind the management are feeling. People have been escorted out for causing outages never to return. While if the change was planned and approved they would have been OK.

For unplanned work you have 24 hours to raise an RFC and 48 before it must be in the approval queue. So as long as its in the system, even if you have caused an outage you should be OK. Generaly unless a complete emergency, people are expected to rasie the RFC at lest in draft, and inform others what they are doing before getting on with it.

I have found if an entire site is down, then generally people know you are working on it so its not like you are trying to hide it away.

Change managemtn should not be about getting aproval, its main task is to notify and insure work one person is doing does not affect others.

shodown

Good

If its used correctly you can avoid catastrophic results.

Case

We were going to do so maintenance on a multiplexer over the weekend which would have brought down quite a few connections to a activity. The activity was fine with it, but a differnet location that connects into it was having the Chief of naval operations there for a visit and would be SOL with no path out. We were able to find this out (didn't know it was the CNO at the time just a VIP coming) early and was able to build site to site VPN's to give the remote activity connection through another site. The day was saved cause everyone paid attentions and change management is used effectivity.


Bad

When it becomes a dog and pony show to please higher up's that its being done, but not being done effectively.

Case

When consulting I was working with a decently sized organization that had multiple call centers somewhere shy of 1000 phones. Change management came out for the server team to upgrade the LDAP servers which were integrated into the Cisco voice solution. The IT director signed off on this without consulting us as the partner on the impact of this. When the servers were upgraded all agents who logged out couldn't log back in as they couldn't be authenticated correctly. So they were't able to do a shift change when needed, While they didn't have a massive outtage hundreds of agents couldn't go home or log off so they couldn't make calls. This was done on a Friday evening, and took around 8 hours to correct so imagine 100's of agents already over there 40 hours for the week getting time and a half for 8 hours.

emerald_octane

the_Grinch wrote: »

What I would really like to see is a system that would detect changes made to various devices and then log it. From there you would get an email stating this change was made and ask for documentation to be completed. If not completed in x number of days, another email gets sent and management is notified. Probably not easy to make, but boy would it be nice.

M$ System Center can do this for Windows boxes, not sure about network devices.

I mean, it can monitor network devices as well, but for instance, System Center can monitor specific server roles such as DHCP, NPS etc so if one of my colleagues decides to change a scope option or adds a new NPS rule, then it will send out an alert to the designated party.
While it doesn't have the functionality you describe exactly, I guess your agents can then hold the change in the queue while they check for authorization, then if no response it can automatically move it to a different queue of "unauthorized changes - no response" for further investigation (which could be sent to management).