Blade Watch

One of the conversations I have been having with colleagues and CIOs is a question that’s been on my mind for some time. What I am about to say is in no way negative and is not to be taken as criticism of the way ahead. There are exciting times ahead for the end user community, for vendors and investors alike. The question is as follows.

As we virtualize and commoditize the infrastructure, the individual components become less significant in a way, whether it’s a server from vendor1 or vendor 11, does the actual underlying tin matter?

1. Absolutely – I have a specific price point that I seek to achieve, that can only be done through our purchasing agreement with vendor8, or absolutely because we want the management software, the extra bits which you just don’t get from any other vendor (real or perceived).
2. No – it will be replaced every few years and besides my servers, my applications are abstracted from it, I want the cheapest hardware financially and operationally.

The problem is that we’re for the time being going forward. (The time being between now and when the data center is virtualized and floats around the enterprise, the cloud or whatever device can power and host it at the lowest cost, a Mac Mini perhaps).  Between now and the future then, data center 3.0 I’ll call it, how do we differentiate ourselves? How do we isolate our servers, our software from the competition?

• Price – but dear me, many a business has gone down that route and volume is key at this point else margins eat your profits – and what about concepts like support costs and service delivery?
• Functionality – great, but dear me reader, functionality is expensive and requires real investment and do our beloved users want to pay for said functionality, everyone says yes until the invoice arrives for approval and sign off
• Apple-ness – by this I mean belief in the product, belief in the service that only Apple seem to be able to achieve in the notebook/pc world – you mean you haven’t got one of these, oh well, maybe next year

I wonder then, going forward as the industry changes, both inside IT and outside the IT industry where the lines are drawn and where go, going forward, is branding enough? Is price or feature competition effective enough as a vehicle to protect revenues. Is Apple’s benefit that regardless of the perceived limitations not that the fans just sit and go wow, look at the design and buy it anyway? How do we then do this in the IT industry? Where commoditization, where hierarchical structures are changing, it used to be that server guy Mike made the recommendations on servers, but Fred who runs trading now wants to know what can be done about this data center space problem, IT is crossing the boundaries that we spent years creating, how then do we change the marketing, the lines of communication and support frameworks to meet this new target audience?

How do we sell servers or software, or a concept to a business unit, a CEO or CFO when concepts of MegaHertz, memory support, or number of PCI slots don’t quite raise the excitement that Mike has when the nice account manager says take a look at this, 5 pci slots and a redundant power supply!

We could go down the services route, indeed, fantastic, would you like  a 2u server or a virtualization product as part of a service? No capex, I repeat No capex, but as the world gets smaller, that poses the old problem, fine I’d love to have my virtual desktops powered by your service, but just one thing, I’ll have it hosted in India, at India prices, not at your UK prices, no they’re too much, and can I have that free mouse mat. I like mouse mats.

We need change, we need progress and development, a move to a scenario where the IT and business merge to create a unified platform for revenue generation, I just wonder if combined the two business units, are ready for this new world as are the vendors. Does this mean a rebrand of IT, a convergence where we offer a range of solutions mixing software and hardware with services and service delivery an overall solution comprising of everything you need, and does that mean the user gets closer or further away from the vendor? We’ll have to see,

I was having a chat with Danny last week asking him about the backup process, I have a theory which applies from the SMB right up to the enterprise. Now at this point I must highlight, this is without formal research, could be seen as generalist, and that for Danny I have removed any references to his organization. Anyway, accepting this here we go, Danny I am an end user, I require data from 2006, a spreadsheet containing some accounts information, and have logged the call below, what happens?

Please can I log a call to restore martins_accounts_yearend_2006, it should be in the backups for March 2006, can you restore it please?

Regards

Martin

A call gets logged to the desktop team, they check the restores and see if the tapes are available. If not then they need to pass the call to operations who formally carry out an inventory of the backup system, establish what tapes are required and then recall the tapes for delivery normally the next day.  Operations then load the tapes and perform an inventory of the tape in order to scan what indexes or files are on that tape syncing it with the backup tape system library.

The tape scan complete (it can take up to 24hrs to index), we then perform the data restore to one of the shared servers which has large drives for this purpose and ask the user to check.

Great Danny, thanks for that, but what’s the lead time?

Oh probably at best 48 hours, unless we have the tapes on site.

With this in mind, does this behaviour not encourage me as and end user to keep everything online? To never archive data that I might just need to keep for future reference? Therefore from an energy efficiency standpoint how much are the backups costing us directly?

An email from Bill about a problem he had with his Compaq DL580 G1, a midrange server that he uses for his PeopleSoft application.

Hi Martin,

I wondered if you could help me with our DL580 G1. I know the server has an onboard array controller, the Smart Integrated Array controller, however we use a Smart Array 5300 to host the array, the onboard controller is disabled.

Anyway, the 5300 controller has failed, any ideas how I can get the server back up? It boots doesn’t load the 5300 controller and goes to a blank dos screen.

Can I just swap the controller and would I need to reload the drivers, will I loose the configuration?

Regards

Bill”

Hi Bill,

Great to hear from you. We’ll answer each issue separately to keep thing simple, and hopefully I will cover everything.

1. You could try using a floppy disk with the array controller firmware on it, or the HP Firmware disk to see if the array controller will work after a firmware refresh – though it does sound like it might have failed.
2. Enable the onboard array controller – providing the existing driver is loaded in the operating system you should be able to boot from that controller and that should be able to read your existing configuration (it depends what the disk configuration is like a disk shelf etc)
3. Replace the existing controller with another Smart Array 5300  is easy, just switch off the server, remove the SCSI cable and the controller, plug in the replacement controller, connect the cable and power on the server. You would not need to reload the drivers. Pressing F8 when the controller is loading will allow you to view the configuration.
4. Replacing the controller with another one, or using the onboard controller should not result in any data loss as the configuration is retained on the controller and the disks.

Let me know how you get on, my number is below in the email signature.

Regards

Martin

There’s an article about an array failure on the Compaq Proliant DL580 here.

This post follows on from the previous one about the BAU and investment split.

What are the benefits?

Removal of excuses on both sides – either resulting from workloads or poor planning/resourcing

Establish centers of excellence for each path:

• Projects continue to develop their own processes and best practice – engineer the build, continue improving the way we deploy servers, allocate storage or network connectivity
• Support teams do not get de-railed from their core task which is limiting outage and disruption to existing user base, focusing on stabilising the existing infrastructure

Establish clear lines of ownership and responsibilities for evolution of process or best practice

Manage workloads and identify issues in the project lifecycle in terms of budgets, resourcing or preparedness

What are the challenges?

De-skill team members in some repsects and can lead to complacency, server is down, but I’m investment, I don’t do support.

Can be inflexible at busy time periods from a resourcing viewpoint – I can only book the engineers I have access to even if Mike in the support team isn’t busy.

It can be easy for layers to occur in the team only Mike does DNS, Bill does hardware configuration and John does operating system load.

Different standards and ways of doing things – what is acceptable in the projects world might not be in production.

Easy for the two paths to go off in different directions due to different goals or objectives – projects don’t do support, support don’t do builds so neither know what the other do and what issues they are experiencing.

I got asked by one the account managers to explain the split within a production team, before we continue we need to take a few steps back. I sometimes have meetings with sales or account managers asking questions about their products or to answer questions they might have to better communicate with their customers.

Anyway let us take the Windows server team at organization 7. There remains two kind of distinct ways of structuring the team.

BAU/Projects split – this involves allocating a set number of engineers to deploying new servers, to assisting projects deliver their technical requirements, and another set of engineers to BAU or Business As Usual, or in English technical support, responding to calls, carrying out changes to existing infrastructure, with a clear focus on maintaining the here and now, doing all it takes to keep the existing systems online and accepting new solutions into support or ‘production’.

Everyone doing everything (no split) – alternatively the viewpoint is that everyone should be doing both, is there a reason that Giles can’t build servers and do support at the same time? It’s 2010, if he can’t should we not find someone that can, I’ve done a few posts which follow about this.

How your IT structures your team will depend on a number of factors, the ratio of support to investment people, will all depend on budgets and workload, and what works for enterprise A wont for B, and we need to understand that each business has it’s own drivers and ways of doing things, ultimately that the way your teams work is core everything else is noise.

Dell.com

RACKNELL, United Kingdom, 16 February 2010 – Dell today introduced OpenManage Integration Suite for Microsoft System Center, a portfolio of software tools designed to help customers efficiently , securely, and cost-effectively utilise and manage standards-based IT resources.  Available at no charge to Dell customers with Microsoft System Center, these software tools enable large enterprises, public organisations and mid-size companies to gain visibility, control, and flexibility of their Windows-based IT environments.

Dell’s OpenManage Integration Suite for Microsoft System Center helps customers get IT assets up and running quickly by streamlining the most time consuming administrative tasks such as bare metal deployments, server provisioning, and BIOS driver and firmware updates.  In addition, these tools give businesses added flexibility in scheduling non-disruptive IT maintenance tasks and help IT staff allocate resources to projects that support business objectives.

“IT professionals continue to look for solutions that help them more easily manage their PCs using standardised tools,” said Bob O’Donnell, vice president for IDC. “Delivering an open, end-to-end solution makes it easier for customers to integrate, deploy, and manage their IT assets. An approach like this is very pragmatic and helps minimise software migration issues while making adoption of Microsoft computing architectures more accessible.”

The News

Dell is making available to its customers, at no charge, a comprehensive set of software tools that deliver hardware management functionality integrated with Microsoft System Center applications.  With Dell’s approach, IT administrators can select and download the management tools they need, helping optimise resource utilisation for their business.  Additionally, Dell has increased the storage capacity of its PowerEdge R510 server to offer an ideal platform for Microsoft Exchange Server 2010 and other data-intensive applications.

Faster Deployment and Provisioning

Through the integration of Dell Lifecycle Controller with Microsoft System Center Configuration Manager and the utilisation of Dell Server Deployment Pack, customers can increase productivity and operational cost savings through efficient, simple, and secure remote deployment and provisioning of servers.  This solution helps reduce time and expenses required to send IT personnel to remote sites for server install and deployments.  Additionally, the operating system deployment process is designed to install the correct drivers in the appropriate directory – saving time associated with preparing update packages.

End-to-End Management for Physical and Virtual IT Environments

Dell provides integration tools for Microsoft System Center Management applications that cover a large spectrum of Dell hardware devices including Dell PowerEdge™ blade, rack and tower servers, MD-series and EqualLogic™ storage products, business clients (Dell Precision™ workstations, Latitude laptops and OptiPlex™ desktops) and network printers. As a result, customers can efficiently and cost-effectively manage their heterogeneous computing environment with solutions that support flexibility, choice and value.

Hardware Optimisation for Microsoft Exchange 2010 Environments

Dell continually optimises performance across its PowerEdge server line to help ensure customers the most productivity out of the business applications they rely on most. Recent tests conducted by the research firm Principled Technologies determined that two PowerEdge R510 servers with eight hard drives and Microsoft Exchange Server 2010 can support up to 10 times the mailbox size while delivering Exchange Mail Server performance equivalent to two legacy HP ProLiant DL385 servers with Microsoft Exchange Server 2003¹. Migrating to Microsoft Exchange 2010 on a Dell PowerEdge R510 solution can also save up to $24,000 in total cost of ownership (TCO) in a year over that same HP system¹.

Anything the vendors can do to reduce operational cost, systems management or agility has to be the way forward. Being able to manage drivers and firmware and use technologies to remotely deploy them can reduce the number of man days needed to keep the server estate up to date, to be more proactive than reactive. An interesting announcement, I’m off to read more.

Silicon

With IT budgets out of the doldrums and – perhaps – on the rise again, analysts have predicted which sectors will spend the most on tech this year.

According to a report by analysts Pierre Audoin Consultants (PAC), software and IT services spend across all sectors worldwide will grow to €625bn in 2010, an increase of one per cent over 2009, compared to the 2.7 per cent slump in IT spending that took place in 2009.

The public sector will be the biggest spender in 2010, shelling out €148bn, an increase of 3.3 per cent over 2009.

It will be interesting to see what combination of best practice, service improvement and investment are put together in order to meet business requirements for cost savings and improved service, do check out the article.

I was speaking with Andy, he was complaining. He is rather busy over in Canary Wharf. The IT team has created a new department to focus on the delivery of their virtualization solution, I was interested to know what was going on, what his issues were, so I did a below the radar question and answer session over coffee, resulting in the conversation below. Andy is a Windows Server Manager, the company is a multinational and all links that might identify him have been removed.

So what’s going on? I ask

They’ve created a new team to take over the virtualization activities and it’s rubbish. They suffer from big head syndrome.

Isn’t that a bit emotional, what do you mean? I reply.

Well we have a few communication issues.

By that I mean they have taken over the virtualization platform, and we’re fine with that, the issue is that they see the VMware and the Hyper-V environments as their layer, they support the virtual machine part but nothing else. So as a user and as a support team working along side, if we get a call for a virtual machine we have to speak to them, but they give us the industry standard statement, “your session is fine, the ESX server is fine, must be a Windows/Linux issue”.  Their list of what they do seems to be smaller than what they don’t, check out this email I got from them they do not look after the following:

• Upgrading or changing virtual machine configurations, including disk space, memory or CPU, that is the responsibility of the core operating system teams
• Decommissioning virtual servers, the application owner, business sponsor or core operating system team will handle the formal decommission and notify the virtualization team when it is time to delete the virtual slice
• Performance monitoring of individual virtual machines and managing system configurations in line with the core operating systems best practice
• On call support or fault identification and analysis will be handled by the core operating system teams and then escalated to the virtualization team once the issue has been identified
• Managing inventory of the virtual machines in terms of capacity and reporting – that is done by the core operating system teams (Windows/Unix)

They and I quote: “Build the ESX and Hyper-V environments, allocate new network and storage connectivity where required, continuing to develop the organizational virtualization strategy “or to the rest of the ‘core operating system team’ adopt the big head syndrome and only do what it is they choose which seems to change on a daily basis.

Are you and the team not being a bit emotional in what you say, any jealousy that you aren’t doing it?

It’s more frustration in terms of ownership. I want them to own the service that they provide, an issue isn’t always black and white, I can’t turn around to a business user and say your server pings therefore its not my problem, and I don’t expect an infrastructure team regardless of who they are to do what I perceive to be the same thing.

There seems to be a reluctance for the virtualization team to get involved in the mundane, the boring stuff, the stuff that in fact pays the bills and actually makes a difference to the end user.  As I look at our queue we have 37 requests for configuration changes to virtual machines. Is there a reason other than we don’t want to that they don’t do this work?

Would they not learn more about the server estate, speak with end users and get a sense of identity? Understand the issues involved and what requirements there might be in the virtual machine configuration? The initial specifications for Windows 2003 virtual machines were 512MB RAM and 10GB disk space for the operating system drive until we requested  more stating that this was not suitable as users were complaining about disk space and memory/performance.

The virtual infrastructure needs to be deployed and supported differently. Your ESX server might work, the network might be configured and the storage fine, but if my virtual machine does not work from an end user standpoint “it’s pants, and I don’t care what you say”. The fact that the team focuses on strategy and ESX server builds means that as a team they do not get involved in the day to day issues, they don’t interact with the end user means when an issue arises we have the same communication issues in which either a unix or windows guy will say this bit isn’t working, any advice, and the standard “ESX is fine, ALL IS WELL”. As a result when we do start having issues with virtual machines, the windows and the unix guys start seeing the virtual machines as high maintenance which feeds back directly to the business, “they said it looks fine… but it’s still broken”.

It’s an interesting conversation and one that I had recently with an IT Director working outside London for a medium enterprise, he had recently turned around to his server guys and said, “Windows/Unix/VMware, I want you to learn all of them, they are all an operating system, a platform that needs to be supported”. “I wanted to remove any concept of alignment to a single platform, there will be people that are best skilled at Windows, Unix or VMware but I don’t want people getting too comfortable, I want to be able to call anyone at 3am, and know that they have enough basic skills for fault analysis” he said.

The team should be looking at our virtualization platform, they know our operational issues, I want a platform that is based on an ideal with a degree of understanding what the issues are and what we can realistically achieve.

I will keep in touch with Andy and note any improvements or changes going forward, how you deliver a service internally or externally can make a huge difference as to its success both within IT and to your end user community, remember it’s only one department, one guy that can change the perception of your service from fine to rubbish, and negativity can become a self fulfilling circle which feeds around your IT, your service and your business – IT is rubbish we’ll never get that working here, so they never try, what revenue or business loss could we be loosing as a result.

Finextra

Don Callahan, Citi’s chief administrative administrative officer, told Computerworld UK that the bank was moving forcefully to hit targets of slashing $3 billion from operational and technology costs by 2011.

The pledge to butcher IT budgets was undertaken by Citi CEO Vikram Pandit in April 2008, as the bank faced up to massive losses from subprime debts. One of Pandit’s first moves was to centralise IT decisions in New York under Callahan.

Callahan told Computerworld that Citi was “well ahead of schedule” with the O&T savings. “We intend to be one of the most efficient operators in our industry, and we have made and continue to make significant progress against that goal.”

An article talking about how this bank is seeking to reduce its IT spend through centralization and improving efficiency, it’s interesting once you look at your costs to see where savings can be made, not just on the technology, but on the operations, the on call. We need to be looking at the way we do business so to speak to identify where our operational costs are, but from several levels for it to be effective, from the engineer to the architect through to the CIO, as all have valid opinions. It was the CIO that told me the story that he was looking through the order approvals and noted that the order included the standard CDROM drive £29 plus the optional DVD, when he asked he got told, it’s always been ike that, the server comes with a CDROM drive but the build is on DVD, so we have to order a DVD drive, fine, he asked, anyone ask if we can not have the CD? Oh I thought purchasing would have said was the reply, purchasing though only ever ordered what they got asked to quote for, the standard server quote included a CDROM. I always at this point quote that rhyme I was taught as a child:

For want of a nail the shoe was lost.
For want of a shoe the horse was lost.
For want of a horse the rider was lost.
For want of a rider the battle was lost.
For want of a battle the kingdom was lost.
And all for the want of a horseshoe nail.

It can be so easily translated into your IT, in so many ways and so many levels. One example I was using the other day with an IT manager for a medium enterprise complaining about his on call costs and ’stability’ of his server estate. I asked him, what proactive steps on your infrastructure do you take? What do you mean he replied, we respond to the alerts we get in MOM?

No proactive steps:

• Clearing down disk space – operating system and data drives, do we need the web logs from 2007 – can they be archived off or compressed, clearing the temp drive, the old hot fixes etc
• Security patching – this by the very nature of the activity forces a reboot, which clears down all the memory and ensures that the server is up to supported revisions
• Defragmentation of the file systems – this can make a big difference to system performance and end user experience – check though if you’re using thin provisioning
• Upgrading the system and component firmware – again forces a reboot and ensures that your system is operating as recommended by the vendor
• System configuration and fault analysis, are we logging failures and looking for patterns, adapting monitoring tools to suit on a per box basis?

Responding to alerts is fine, but you get what you put in, even a legacy server estate running NT4, Windows 2000, can be made to run very efficiently and with a degree of stability, what you need to do is focus on the core, the server platform, standardizing and bringing everything up to date, just doing this and combining with a “health check” (disk space, memory, configuration) can make a world of difference to your on call, to your support costs and your end user perception.

Finextra

Last year saw a decline in IT spending growth, from 3.1% in 2008 to a mere 1.7% but Celent predicts a steady uptick over the next few years, with the figure reaching $55.2 billion in 2012.

Celent says 2010 will see spending on total new investments grow by a solid 7.1% compared to an 11% fall last year. The overwhelming majority of this spend will be in wholesale banking, with an increased focus placed on corporate cash management as banks look to upgrade their ageing platforms to woo additional business.

In contrast retail banking IT spending will grow just 0.5% in 2010, compared to a 1.2% rise in 2009, says the research firm. Cuts to retail banking operations and an emphasis on self-service are fuelling the trend but the pain will be short-lived, and the start of a turnaround is expected in 2011.

Another article discussing IT spending in the near future, different sectors will of course see different results, regardless focussing on the task at hand for many a CIO will be the key requirement, whether it’s answering calls within your SLA, refreshing the hardware to reduce support costs, or deploying servers within the time constraints placed upon them by the business. We need to be changing the IT spend away from support towards investment resulting in lower long term costs, delaying the investment can work out more expensive than refreshing or replacing the hardware and software where possible. I wonder if we will see any development with this in the ITIL framework in terms of chargeback/service delivery or even concepts of marginal cost – if we had out true marginal or operational cost, could IT be in the position to put together more effective business justification plans for investment projects?