I was having a chat with Chris the other day, he was telling me some news about work. IT had recently implemented a new rule. It was known within the IT team as the three year rule. It basically meant that IT would no longer rebuild any x86 server that was over 3 years old, it is linked to their programme to virtualize the infrastructure. There are exceptions to the rule (obviously, rules are rules, but you need to be flexible to the business need/case). Anyway the following conversation resulted – I’d like to thank Chris for dinner at Black + Blue (where I got a fantasitc Vegetable burger), it’s a wonderful place just behind Selfridges. Anyway I asked him a series of questions, I may have paraphrased and I’ve removed any references that might give away his identity.
So what’s the three year rule then?
Our servers are purchased and depreciated over 3 years. With that in mind to help reduce our goldstock, our energy consumption and support costs, we’ve implemented a rule that we will not rebuild a server that is over three years old.
Sounds a bit harsh, are their exceptions?
There are exceptions, if a production server failed and required the operating system be re-loaded or restored then obviously we’d rebuild it and get it back online. This is aimed at project type activities, to prevent old hardware being re-commissioned and used by the development teams or re-allocated between business lines.
Who decides the exceptions?
To avoid debate the rule applies to IT and Customer systems, the concept therefore is that IT can’t rebuild that DL380G1 as a domain controller, equally, the end user, that business unit can’t ask IT to rebuild that server. The server is either replaced with new hardware or a virtual machine provided.
Why implement the rule?
The new standards were implemented for a mixture of reasons. Our primary data center is nearly full, so we’re having to move more back office/less important systems to our other more remote data centers. We’ve been finding that the older systems are less flexible when it comes to remote management. A Compaq Proliant 2500R has a RIB card, meaning I can power cycle it etc, but I can’t easily mount a cd image to copy that security patch, those system updates or application code, making releases/system upgrades a longer term process, particularly for customer facing systems. This becomes increasingly important as we move some systems to our other data center, we can’t afford to loose an engineer for half a day to mount a cdrom.
Additionally we were having issues where a server would go live and then have a hardware failure causing disruption to the business team and an extra support overhead. For example, an application team requested that their DL580 which had been decommissioned from production have Windows re-installed on it. The team had the risks highlighted to them, but the risk was signed off. The server went live and within about 6 months the array controller failed and the system board had to be replaced. It’s not something that the diagnostics could have predicted at the time, but for the cost of a new server say £1000, the business unit had loss of service for three or four hours; whilst an engineer was called, the part re-called from stock, the board replaced which requires the server be completely disassembled. Additionally as this new system started getting more users, the application team were forced to purchase additional memory which proved to be expensive due to the age of the system, re-deploying a system therefore cost more than buying a new one.
We’re also seeking to reduce the cost of the goldstock, we have 12 types of Compaq and HP Proliants spanning legacy Proliant R type servers, up to G4 servers, all have different components some of which we need to keep in goldstock, there are common components, disks, memory etc, but if we reduce this to 4 servers or even 6, we’ve reduced the number of components we need to keep in stock (and therefore the cost of the hardware support contract – removing those old systems allowed us to negotiate the hardware support contract more effectively).
It also allowed the build team to focus on supporting a set number of model types, rather than consider and test the driver pack on 12 types of server, it’s now 4, which reduces the time needed for testing/evolving the operating system build, and allows us to get more up to date with our support of the operating systems. At the same time, as we make the server a fixed life span, we can also introduce the concept of a fixed term operating system, start talking about getting rid of Windows 2000? As an operating system it’s fine, but we’re looking at Windows 2008, and we’re eager to avoid supporting Windows 2000/2003/2008 application, web, file and cluster servers – particularly for security patching/system administration/driver packs etc.
What’s your viewpoint?
I’ve got mixed opinions, the three year rule will allow us to get rid of those systems that are ‘high maintenance’, those Pentium III Compaq Proliants (or older) which we don’t really want to support for a mixture of reasons. It does mean however that we get rid of servers that are perfectly adequate for the task, but then I suppose they could be virtualized, it also makes the server appear more expensive as the hardware needs replaced more regularly.
What has been the reception positive/negative?
We’ve got a mixed response. Some application teams aren’t that bothered, many have found that a virtual machine works fine for them anyway, so us taking their DL380 or Compaq 6400r from them isn’t that big a deal. Those business lines more aligned to the trading floor though have tended to be a bit more assertive – the main debate has been the cost, that a DL360G1 works, it can run Windows fine, why should they buy a new box, at this point we show them the ‘three year rule’ justifications document, discuss with them and talk about the options for the way forward – sometimes this means getting sign off to have it rebuilt anyway. It’s all about the way you approach it, we were careful to get stakeholder sign off, so architecture from infrastructure (IT), the developers, and the business sponsors got their own justifications statement so we could get buy in for the ‘rule’, or standards.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
Blog videos