When is a change not a change asks Chris

I met Chris for dinner this evening for a chat, I was asking him how things were going and he then told me what’s been happening. the following is an extract of the conversation to avoid any references to his identity. I asked him to step back and the following is the result of the conversation, as ever Chris remains ‘below the radar’, he works for a bank in Canary Wharf. I’ve put his replies in italic.  Chris mentioned some issues regarding the change process which I thought were worth noting and talking about.

What’s the issue with retrospective changes. I ask.

We’ve been having a debate with the change guys. At the moment we use the change process like any other windows server team in any other bank. The debate has been surrounding a retrospective change.

A standard change is one where we announce for example we’re adding disk space to this cluster next Sunday, a retrospective change would be, I rebooted the server because it was failing, or we needed to resolve an application fault.

I see, (I reply). That’s fair enough, what’s the issue then?

There’s been some debate about whether the retrospective change should also cover restarting a service, clearing down log files, restarting the web services, SQL for example.

Let me give you an example, I restart IIS on a web server because the site is down, this technically is a change to production, but the work would traditionally be done under and incident, site is down, restart IIS, move on. The change team and the business lines are asking “Is that not a retrospective change?”. Should they not be notified?

It’s one of those things which is going to depend on your company change policy, coupled with your business teams requirements, that is the sensitivity of the system involved. Ultimately we need to define the parameters of the change process. Restarting the services on a server could be constituted as a change, by logging in you’re changing the server, you’re writing data to your profile, your registry keys. What we need to accept though, is that it’s not always going to be black and white. Should you not be operating on the basis that if you’re restarting a service/process/server to resolve an issue is only a change if you are making a change to the systems’ configuration or causing a further outage to the end user(s)?

I see, and what’s the VMWare issue?  I ask

Well that’s an interesting one. We’re VMWare’d up, we’ve got VMotion to allow us to move virtual instances around the environment so that we can carry out necessary maintenance on an ESX server, or if one application needs to do some special testing, we can move their machines to a server with less virtual machines on it to avoid affecting other users.  The issue is moving a virtual machine from ESX to ESX a change? 

Also we’ve been adding new networks to some of the ESX servers to bring in new services, market data feeds etc, is this a change?

The answer I suppose is yes. But again it depends on the context. I’d argue that moving machines around the infrastructure is a change but not necessarily one that’s going to result in an outage to the end user. Crucially will it not depend on the chargeback model? If I own the infrastructure, you’re effectively renting a process and some storage from me with a requirement for a specific availability, the processor, the asset that it is powered by should be immaterial. Certainly as we take virtualization to the next level and we try an automate the virtual infrastructure, surely this kind of thing might be automatic how do you manage that in terms of the change process.

Stepping back a minute if I decide to shut down and replace the array controller in an ESX server during the day, and this means I have to move virtual machines to another ESX host, I suppose this would be a change as you’re carrying out scheduled work. Again it’s going to depend on your business, your change process.

Both issues are set to continue, in the meantime thanks to Chris for going over what he’s been talking about, and for dinner.