Big Data in three stages

I was having a chat with Mike, a colleague who is not a believer in Big Data, he is somewhat sceptical about the concept and was asking how I see Big Data, in essence “ok so explain to me where this Big Data thing is coming from and how you see it becoming mainstream, I don’t get it”

Well this is where I step back for a second, using my left foot in the here and now, the right in the future as ever. Big Data is going to mean a different things to different people, and to different businesses dependent on where you are. But it’s all about recognizing the challenges of the growth in data capture and usage, as well as empowering end users, business users and technical teams to capitalise on the data we have in order to create opportunity for revenue generation, savings or empowerment.

In order to be ready for Big Data, we have to recognize, remediate and innovate, recognize where the issues reside in terms of our data management, remediate with the right set of businesses processes and technologies, innovate by using the data we have to create value and business transformation. I have noted some bullet points and comments that I made to Mike in this space below, it’s a work in progress, google Big Data and you’ll get a range of comments and analysis, what occurs to me on a basic operational level is two things, we can absolutely allocate you hundreds of GB of storage space, TB of space if need be, and we can manage it so that it’s working to peak efficiency, I am not however sure of the following:

• That the volume of space allocated is actually needed both by the application and the users that are using it – efficient data efficient code and application architecture is needed to avoid duplication
• That we will be able to restore that data within the timescales you expect, particularly as we scale up to millions of files in a file system

Stage 1 – Agile Data Platform

Move our legacy storage practices, technologies and procedures to be data centric rather than infrastructure centric. In the traditional enterprise the business teams or application request for example 2TB storage for their SQL server and 300GB for their shared departmental drive on their NAS solution. However there are three main issues around this scenario, firstly data growth continues at a fantastic rate, investment is not always what it should be therefore users request more storage than they typically need (often as they don’t know) creating data waste, secondly not all data needs to be on that tier of storage and more importantly needs to be online, thirdly the management of the different storage platforms is often different requiring a different set of skills and management tools.

We therefore need to storage generation two, where we have online and offline data, centralized management based on need and on usage, using simplified management tools where possible to move towards a managed storage offering from which we can then derive value.

Stage 2 – Backup and Recovery on demand

Move our data strategy towards user data centric protection and recovery, so that we can have a unified portal for backup and recovery, fewer tools, fewer management platforms to enable swift and effective data recovery on demand even via a self-service offering subject to data privacy and administration issues.  We need to be able to understand that if I want the month end of June for 2009 data for our payments platform that we know that the backup ran, that the backup wrote data, to which tapes it wrote to etc in order to deliver the opportunities for restore.

Stage 3 – Analyze our data in order to derive value

Being able to analyze our data in real time on demand, to meet different user groups needs can be transformational, whether it’s marketing being able to manage social media, or the sales teams being able to monitor transactions and identify opportunities for follow up, being able to know your data, your customers and your position is an transformational offering.

Big Data is a concept, it encompasses many features each have different levels of relevance to your business, key for you is to understand where it fits and what the result of this could be, whether it’s more servers, more complex applications or demand for ever more storage. What matters is that as we move to an information age, one in which social media is not necessarily the center of people’s world and decision making, but it is influential and can join the dots between the consumer and the business, the charity and the donator, being able to manage and analyze your data can bring you closer to your customers, help identify opportunity and in doing so create value and opportunity.

This site uses Akismet to reduce spam. Learn how your comment data is processed.