Bladewatch.com

I had a chat with a service delivery manager who was telling me how he was getting on in their enterprise. They’ve been talking about deploying VMware, but before they did this, he was wondering if we should address some communication and perception issues.

He provided the example below, which illustrates brilliantly the kind of environment and relationship between the infrastructure and application teams. The underlying lack of ‘trust’ or working relationship other than email constituted in on going issues of delivery, innovation and ongoing process innovation. That the hope of deploying virtualization or consolidation might help with this (from IT), without understanding that until the trust, that there is effective communication, no technology is going to be the fix.

Related to this was one of the application teams logging a call to have their development server have the Oracle client installed. It was great to see the call log and the email exchanges. I’ve summarized the emails and made them generic to hide the organizations name:

“Please can you log a call to the Windows team to install Oracle 10.2 on development server NYKW90001 and email when complete.

Regards

John

Settlements Development”

The Windows engineer saw the call, scheduled the SMS drop for 6pm (London time as the call was Logged in London) for that Wednesday, which installed the Oracle client and then rebooted the server as it had to load some system variables and bank specific settings. He then updated the work log:

“Drop scheduled for 6pm and completed ok. Emailed user, closing call accordingly.”

A complaint was raised by the Head of that application team, the email as follows:

“Please can IT explain why the server was rebooted, also confirm who authorised the reboot of this server as this caused disruption to the development team, which is an unacceptable business expense and service disruption.

Also please note that from now, development servers for the Settlements application cannot be rebooted without prior authorisation, a service request and email from me or a manager of the team, preferrably with sufficient notice and out of core business hours.

Mike

Head of Settlements Development”

Windows Server team leader responds:

“The server was rebooted in order to complete the installation of the Oracle client as per request 832491 logged by John, the call did not note that the server could not be rebooted, therefore no further action or notification was taken.  I suggest that the development team note if there is an issue rebooting the server when a package is to be dropped.”

The Head of Settlements development responds, copying in all development team line managers

“Thank you for your comments, IT does not follow any level of due diligence in preventing outage? This outage caused our team disruption and development time, the development platforms remain an important element of the IT infrastructure in order to provide continued enhancements requested to the business.

I do not expect that my developers know the contents and implications of deploying application packages for infrastructure packages.  I suggest we look at creating a process to prevent IT unnecessarily causing an outage to business systems as a result of the work they are undertaking.  Maybe we need to look at a Development change process?”

As a result of all the development managers and development teams being copied in, a debate results about the fact that IT might just reboot any development team whenever it likes without debate, that they don’t understand the implications and maybe that development needs to be re-defined and have its own change process. The actual outage resulted in a delayed development batch causing the development team to get their results 45 minutes late. IT then became defensive as the discussion continued, allowed the debate to continue via email (a media that is all to easy to skim read and press send), resulting in:

“All requests to the Settlements server infrastructure logged, need an attached email word document with authorisation from the Head of Settlements Development, with a date and time when the work is to take place.  An email then to be sent to the Windows team notifying them of the work, and the work to be completed in the scheduled time, and an email sent to the development team when complete.

No work to be undertaken on the Settlements development infrastructure without permission during the business week without 96 hours notice unless mandated by the development team.”

This creates a substantial management overhead and has to be replicated to business teams requesting it, who then site Settlements as the justification, it results in changing development as an environment to become effectively Production in all but name.

In effect a 45 minute outage on a development platform resulted in several man hours spent across business lines debating the issue, debating responsibility and resolution, resulting in additional overhead and disruption to service improvement or platform evolution.

What should have happened.

A phone call to the Head of Settlements Development, basically outlining that we apologise that the server was rebooted, that the server was rebooted as a result of a request from one of the development team and we had thought everyone was aware that the work was going ahead. Of course delivery and availability of the platform is our key objective, and we will work to ensure that the team are notified of the importance of the Settlements infrastructure, and that to call the team before undertaking any work to prevent any unexpected behaviour or outage. This typically involves an email to the Windows team outlining the importance of the Settlements infrastructure being available, and that any changes be notified to the Settlements Development team before being undertaken, unless otherwise scheduled through the change process.

The key to this is that if I see enough emails back and forth saying someone wronged me, they become gospel, we need to abstract debate, that way it’s a mistake by whoever, it’s fixed now, next.

Crucial:

• End the debate.
• A phone call transforms the conversation from one involving the universe, it avoids any debate, of you guys, to a what pains you and how can we fix this.
• The longer the email chains continues, the longer we are stuck in the past and not moving forward, we’re in the here and now what can we do now to prevent it going forward.
• Address each point in a binary fashion without debate, apologize where necessary, adopt the, it’s happened (not being flippant), apologies, we will do everything we can to prevent, next!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.