Bladewatch.com

Chris called me up in somewhat of a panic down in sunny Canary Wharf, he had a problem. He had opted for some easy weekend work, he had to perform some system updates to the production VMWare farm on their servers, everything was going to plan.  He had failed over the virtual machines to the other servers and was applying the individual configurations and system updates to the server before rebooting.

The following is a transcript of the call he made, I have removed references to the vendor to avoid any conflicts of interest or emotional outbursts.

Thankyou for calling Vendor8, your call is important to us, this is Janet speaking, how can I help you.

My server is broken, I have rebooted it, it’s on the bios screen saying the controller has detected changes, do I wish to accept these changes or reject them. I cannot loose this server or the data on it, can we call out an engineer or tell me which option to press?

Sure sir, we’ll get there, not a problem, can I have your name please, your company name and address, your system serial number, system, array and lights out firmware, the operating system, the exact error code.

Chris Surname
Large Corporate in Canary Wharf with nice sunny office and trees/plants inside
System serial number is 999999999 and model is 494940
Firmware 1.2, 4.3 and 1.11
POST Error 2781 detected please press enter to accept changes or escape to reject

We have had this before and ended up calling an engineer as when one of the guys did it, the server died.

Not a problem sir, your call number 1082319 has been actioned. Now the screen you’ve got, at what point of the POST start up is it, and what happened.

The call goes on, Janet was following protocol and industry standard best practice. The result. Janet suggested an array firmware upgrade to 4.32 as this fixed a known problem and will prevent further issues.

The final result, Chris thanks Janet and says he will absolutely do that and call them back. He then calls me vendor8 are rubbish, I hate their servers, so my server says error 2781 detected or some nonsense, what do I press enter or escape?

In what context I ask, what you been doing to what? So he discusses his previous actions and the server that is reporting the issue, I deploy google and before you know it, find a blog post quoting his exact error and issue saying ‘just press enter’ it’ll boot fine oh and when you’ve got a moment check out firmware version 4.32 which will stop it doing it again.

You see it’s not that support were not doing as they were supposed to, they were fulfilling their obligations in line with their business processes and preventing unnecessary engineer call outs, however Chris was in a situation of liability damage, he wanted a binary answer not to permanently fix the issue, he just wanted to know how to get his server to boot. You see his change window was for three hours, and he was completing one more reboot before handing everything back and putting everything back as it was. Significant deviation from that plan would require additional sign off and crucially management intervention. Therefore returning everything to service was the priority, a permanent fix was not deemed in scope, he could arrange time at another date to fix the issue properly.

Share and Enjoy

http://content.dell.com/us/en/corp/d/press-releases/2010-3-16-poweredge-westmere.aspx

Dell has announced a refresh and enhancement of it’s PowerEdge servers featuring the Intel Xeon 5600 processor. This includes their blade M610 and M710 servers, their R410,R510,R610 and R710 rack servers and their T410,T610 and T710 tower servers.

It will be great to see what improvements have been made to the platform, there are some significant energy savings and performance, combined with improved focus on the efficiency of the power supplies, on board hypervisors and AES-NI to further adoption of encryption for security.

Their focus seems to be around ease of deployment, reducing the barriers to succcess and improving systems management for which they must be commended, anything we can do to make it easier to deploy and support the server platform has to be a good thing, whether it’s in terms of common drivers and firmware, or more effective deployments around their Lifecycle Controller. I’m off to read up more.

Share and Enjoy

HP have released 16 new servers based on the new Intel Xeon 5600 processors, which brings enhanced performance and efficiency, with further optimization for virtualization solutions. They are quoting impressive savings above their previous generation G6 servers, with 27 times improvement in performance per watt, featuring technologies like their HP Power Advisor which can improve energy efficiency by 16 per cent and discuss the sea of the Sensors, their strategy of effectively understanding what the different elements of the server are doing and adjusintg the power and air flow accordingly. It’s always interesting to see how the new servers compare, what new features both around the hardware, and the management software have been improved.

I’m off to read up more, it’s interesting to note that the new servers comprise of their blade and rack servers, very cool.

Share and Enjoy

I was speaking with one of my CIO friends who works for a multinational in the city and asking him about a problem he has been having. I’ve removed anything specific to his business, but his problems remain the same.

“I keep telling the guys and the VMware team manager to up the virtualization ratio, we need to be virtualizing more systems, and I don’t understand why there is a perception issue for a few teams, what the challenges are., why we aren’t getting there more quickly, I keep getting told of dependencies and issues, of we’ll get there but..

I’ve got a direct target from the head of business to have virtualized 70% of the server estate, comprising of production and development overall, they just want to be seeing progress, so I have been working with a small team I created to simply do this. The idea was that they would focus on virtualization activities, they wouldn’t get caught up in the day to day stuff, and could therefore start arranging and completing p2vs, that way we could prioritize old systems and retrospectively and at the same time identify current and future projects to see what could be operating in a virtual machine.”

Anyway, we’ve managed 40% and we seem to be slowing down the pace, and it’s only when I asked the manager into my office to explain what is going on that she identified our bottlenecks.

I was disappointed on several levels. The team had:

• Started making decisions on what was a supported disk configuration or was deemed as best practice within their matrix
• Operated on a basis of the age of the hardware rather than the workload and the application
• Neglected to effectively communicate the business drivers, when I asked to see their intranet pages it was full of sales pitches and technical stuff, nothing about why our organization was deploying virtualization what we were seeking to achieve.
• Had broken the virtualization estate on a business line basis, which was fine idea in design, but I now had a problem that the trading ESX servers were ‘full’ but IT had capacity, however IT didn’t want to put trading on IT as that meant the systems had to be more reliable/avaialble reducing testing for upgrades or new configurations. It also meant that we had idle capacity for different business lines which could not be shared for whoever needed it.

We had to turn everything around. I also asked them how we were answering the billing mechanism, there was no response, guys come on, how are we answering people’s questions about depreciation and support costs? How are we getting people to pay for the environmnet, £100 internal chargeback is better than nothing whilst we deploy the technology and brainstorm billing methods. Because we didn’t have a billing method the guys had got stuck around the business line basis, because it was easier, if trading needed more capacity we asked them to buy more ESX capacity and licenses, not surprisingly their response was, superb idea, “come back in 2012 just after the Olympics, we might just be interested then, in the meantime, can I have 16 new physical 1u cheapy servers, and can you rebuild that DL380 G4? It’s trading so you can’t say no”.

An interesting issue and who’s at fault? The CIO let me speak to his virtualization team manager over coffee with his knowledge. I got labelled as a server consultant, a vague enough title for her to be able to rant her problems and challenges so we could find out what was going on, I reported the results to them both and have summarized them below:

The Management

• No clear definitions of what is in scope – what the priorities are
• Lack of billing method to establish how we pay for and fund the environment software licenses, storage and hardware
• Lack of day to day operational interest – the problem of logging a call and thinking it will fix itself
• Procedural and process based problems – virtual machines were built in the same olden days way as a physical box, reducing the benefit of the platform, and the CTI for issues relating to virtual machines was not defined – if a virtual machine went of the network was that a virtualization team issue or a Windows/Unix one, in the meantime the call got passed around the universe and the issue not resolved – frustrating the end user community.

The Virtualization team

• Worrying for everyone – when asked what the policy was if a user asked for 400GB storage, they replied that wasn’t supported, the user could have 40GB and request more when they needed it. Otherwise they would quickly use up the storage that the organization had.
  • – That’s not their issue. Their issue is to identify barriers to achieving the 75% which is storage so that the management team can either set policies or buy more storage
  • - In doing so people with larger storage requirements opted for a physical server creating workload later, and also damaging the virtualization concept – they’ll only give you 40GB don’t request a virtual machine
• Lack of branding – they had an intranet page it had we use this VMware version, we use these kind of rack servers and these networks are supported
  • - No business benefits such as agility or ease of upgrade/changes and rollback
  • - No best practice as to how to request a virtual machine and what specification of machine is needed
  • - No success stories – mini succeses kind of things, we’ve reduced IT from 300 servers to 11 physical ESX servers and 20 physical servers running a mixture of Windows and Linux for the backups and the databases
• Waiting for dependencies – the CIO asked how many of the 37 networks were supported, 11 he was told, why? Because they were waiting for networks to complete their vlan tagging and specific network upgrades rather than request that network, this ruled out complete environments and applications comprising of development, staging and production systems which might have been or were happy to be virtualized but could not because the network was not ready yet.
• Clustering and high availability – too much focus on high availability not in getting the job done. The team had actively held back on virtualizing systems that had networks or specific requirements and could not be put on the high availability virtual environment. The point – they hadn’t asked the user base if they wanted or needed this functionality, not every system is tier1, not every user would necessarily need or want this function, as long as they know if broken it will be available in 2-4 hours that meets 75% or more of their end user base needs.
• Inappropriate candidates selected for virtualization and missing out on others, selecting multi socket, multi core batch processing low latency systems and choosing them saying it should work without clearly defining the deliverables or the rollback procedure resulting in delays and virtualization rubbish message being distributed around the end user development estate. At the same time, infrastructure servers were often left or not ruled in scope for mixed reasons, and systems left out of scope were not highlighted for a hardware refresh, re-design or requirements identification. The best example being their tape backup system for a separate external environment comprising of some legacy 7u rack servers, it works and it will be decommissioned soon, rather than identify the savings achieveable through refreshing them with a 1u server and tape backup solution. This further reduced savings or possibilities – was the tape backups system even needed, or was it an inherited thing that was there and nobody knew why they had it.
• Everything and I mean everything was stored, published and updated in the virtualization world. The CIO got no view of what was happening, he didn’t know that the trading team had recently ordered a new server simply to store 500GB of data, because a virtual machine could not be supplied with that much space – his response, why isn’t it going on the NAS? Who scoped this much storage? What’s the archiving in place, and how important is it?

Share and Enjoy

Cisco have announced some impressive results in virtualization benchmarks using VMmark on vMware on their System B250 M2 rack server, with a focus on server consolidation and reduced total cost of ownership. They have also announced impressive benchmark when hosting Oracle WebLogic Server application as well as great results in the High Performance Computing benachmarks Specfp_rate_base2006 and others.

Well done to Cisco for achieving these results, it’s always cool to see what can achieved with the right combination of configuration, layered components and software, as with anything these benchmarks are going to be specific to that environment. As with any benchmark how a system operates in your environment will depend on many factors, from the network, to the storage, to the server, the operating system and the application. That you have all optimized for the platform is key, even simple things like firmware, and the hot fixes or service packs or system configuration can be the difference between impressive performance and optimized. Regardless, I’m off to check out these results in more detail.

Share and Enjoy

So we are (thanks to Dell) stateside visiting their ‘Solutions for the Virtual Era event’ in San Francisco, during which it will be great to meet with the guys from Dell and see what they’re talking about and see their new servers. I’ll be posting pictures on our flicrk account and twittering/blogging throughout.

As ever my phone will be on and I’m here until Thursday morning, so if you wish to meet me or have a chat whilst I’m in sunny San Francisco, do email me Martin237@gmail.com and I’ll call you back.

If you’re at the event and would like to talk to me, do get in touch, I’ll be the one wearing dockers and my infamous tan Trickers boots.

Share and Enjoy

It has been a busy few months for a number of reasons, the blog has been re-designed, we’ve been talking as a team about the direction of the content and scheduling more interviews, more content.  At the same time, a change of jobs on a personal level means I’m as busy as ever, that said the blog continues, with a break for the past few weeks. To all those that have been emailing their announcements, I’m working on them and am scheduling them in with a few of our interviews, thanks for keeping me up to date.

So what has been on my mind recently having spoken with managers/CIOs and server guys? A number of things, the middle east, India and China seem to be areas of real growth and excitement, in the city bonuses, salaries and job security are all on the radar (rightly or wrongly). There is also a sense of change of mindset towards acceptance of virtualization, but when I say that it’s not that we never accepted it before, it’s more that the way we procure and supply elements of the infrastructure is changing, concepts of saying to someone ‘that’s not core, give it to someone else to worry about’, which might refer to backups, to the email system or the HPC solution we use, are becoming more acceptable or at least discussed, that I might ask HP or Dell, or IBM or a service provider to supply me with a grid in a box solution, or buy the capacity or services I need rather than specifically deploy the infrastructure and support it myself. We’re not there now, but we’re talking about it, and this leads to possibilities, and further talks about the commodization of the infrastructure which can only be good and bad for the vendors at the same time. Good in the respect that I might now find more opportunities where they might not have been ‘We only buy IBM’, might not be as set in their ways, well what can you Dell, Cisco, HP, Oracle or Fujitsu (to name a few) can you do for me?

As a geek, as a blogger and someone who likes to watch the industry (whether it’s the car, the business sector) or IT, it’s interesting to see the convergence of the different elements of the IT service and infrastructure markets. What does it mean going forward?

The way I see it, we have to change two things. As a vendor I need to become more vendor neutral whilst maintaining my own indivuality and USP or brand you might say. Turning around and saying no we don’t work with them isn’t good enough, as an end user we will find someone that does. What is increasingly important which I may bore people by going on about is WHAT IS INCLUDED, NOT WHAT IS NOT. It’s wonderful to hear the vendor sales guys compare producsts, compare features ‘their one only has four onboard network ports’ etc, but what they are missing is the concept of what use, what individual need that product fulfills. We need to manage the pride we have with our products, our business with linking it to the business need.

More and more, going forward in the enterprise space it’s compatibility, it’s lowest operating costs, the absolute cheapest way we can do it to achieve a specific need. The concept of change can be world ending, the G6 uses a different array controller, can’t do it, it’ll take three months for Jim to change the driver in the build.

For the SMB they will all say it’s about money, however linked into that is the ability to call someone at 11am, at 3am and say it’s broken help me, that’s where all the non-chargeable bits come into play. How the support concept works, how you log a call, how you are treated, what benefit of the doubt is deployed, the ratio of computer says no, mixed with the ability to google the technical message on screen all transforms the end user experience.

So how do we bridge or converge (the buzz word I keep hearing) these two markets, and how do vendors approach their customers going forward? I think we need to reduce the technical comparisons, the my blade is bigger than yours, focus on two things:

• Listening and understanding your existing customer base
• Spending your existing advertising dollars on a mixture of support and end user fulfilment

What does this mean in English?

The reason the Bladewatch team use a mixture of D430s and D630s which we bought from the Dell factory outlet?

• I can get the parts from Dell / Ebay / Reseller easily and quickly
• They are industry standard – it just works, whack on the operating system, download the service pack, a few drivers and leave Windows Update to do the rest
• It comes with the cds, there’s no nonsense stuff installed, the closest PC I can get to a MAC in terms of simplicity and ease of repair

In the server space, then how could we achieve the same thing for team MacLeod or team Bladewatch:

• Real world content and scenarios for configurations or support issues – I don’t speak server language at the low level, error 238 that my array controller memory cache is preventing the server to boot, is it accept or reject changes I want – that’s what I need
• Real world support – the ability to quickly and effortlessly ask a question without logging a call ‘Ask Mike’ – I have a brilliant interview with a server guy about this
• Community around the business and the technology – understand again what pain points, what issues your customers have and educate them, with the usual end user disclosure statements, to swap a disk follow this video, but if you aren’t sure call an engineer
• Make the support mechanism your identity – just look at what Apple did with their genius model – where’s IBM x3650 or Dell 1950 guy when I just want to know what option to press and how much would your organization save by me not logging a call?
• Look at how we charge for parts or make them accessible – this is always a difficult one, but I wonder how much loss of revenue we’re missing on because the official vendor price for that disk is £89.99 but I can get one from reseller7 for £39.99 – granted we might not be a component seller in business models, but this is what alienates customers at marginal cost.
• In essence less formal documentation and more issues based on fixes and get it back online scenarios – it says press F1 or F2, which is the right one to get it to boot and rebuild the drive, which one causes it to go on its holidays?

You could argue that as the virtual world takes over, as we switch to virtual data centers, running virtual applications on virtual machines using fewer physical servers, with virtualized networks and storage, the server becomes irrelevant. Whether it’s a Compaq DeskPro, an Oracle  industrial strength SPARC based server or a rackmount from a local guy, it’s easily replaceable. But there are two interesting elements to this. As I reduce my server estate from 1000 servers to a few hundred, even thirty if we see the latest announcements in terms of performance and scalability, do they not become even more core to my business. How they all act together, how I manage the system health, report on the inventory and identify what is running what, what the impact of a change might be, what diagnostic and support tools become even more important.

At the same time, it’s so easy to forget that every business, big and small is in this state of operational bliss where everything works, where everything is virtualized, with per cpu/GB billing, where we have no problems and the only consideration is the age of the hardware and reducing the support costs. There are still many businesses trying to cope in the BAU world (Business as Usual) the support, the here and now where, it really would be nice to talk about that, but we have 400 support calls and requests, we have 96 servers which need hardware components replaced or hardware diagnostics to be scheduled. Where we don’t get immediate budget to replace hardware, where I might take that DL380 G4 server and swap it with my DC which is a DL360 G1, because it’s newer.

I wonder therefore if we can adopt a policy where no one gets left behind in an IT sense. Where we share ideas and concepts in doing so earn revenue and create opportunities for end users as a community and the vendors at the same time.

Share and Enjoy

http://www.finextra.com/News/Announcement.aspx?pressreleaseid=32725

The application allows businesses to accept secure card payment transactions through their iPhone.

John Clarke, Head of Product Innovation commented, “The iPhone Virtual Terminal is designed to process credit cards securely over the 3G or WiFi Network and is compatible with Apple’s iPhone 3G or iPod Touch technology. The product supports the secure processing of card transactions through major domestic and international payment processors such as Elavon Merchant Services, AIB Merchant Services and Barclaycard International and is available now for download from the Apple iPhone App Store.”

How cool is this? The more accessible both in terms of technology and in cost that we can bring to the small/medium retail business, the more opportunities for revenue and commerce, I’m off to read up more.

Share and Enjoy

Check out these podcasts/videos from HP, they cover some interesting content and include coverage of the HP BladeSystems day, very cool.

It’s always interesting to hear what people are talking about in the IT space, what their drivers are, the more we discuss what drivers and challenges we have the more we can think about the right solution for my business, whether it’s blades, rack servers or a desktop, that it works is all that matters.

Share and Enjoy

http://www.pcpro.co.uk/reviews/servers/356029/hp-proliant-dl4x170h-g6

Blade servers have always been the best solution for maximising rack space, but they could be about to lose their monopoly. HP’s latest DL1000 rack server series delivers four server nodes in a 2U rack chassis, and in this exclusive review we take a closer look at the new DL4x170h G6 quad-node model.

The DL4x170h offers a good range of storage options, with the review system providing eight hot-swap drive bays. Options are available for two eight-drive SFF bays that slot in at each end of the front panel. A glance at the back shows each node comes complete with its own power button plus monitor, network, serial and pair of external USB ports.

An interesting review of the HP Proliant DL4x170h-G, a server which has four nodes in the one 2u form factor. I do love the concept for example in a small compute farm or even as a small scale virtualization solution? I am off to read up more about them, but do check out the review, a mini blade solution in a 2u form factor if you like.

Share and Enjoy