Just mulling a few things over about Big Data and the rush over the last few years for people to claim that they have 'Big Data'. I think the message has been missed and is yet another bandwagon (for most people). The definition on Wikipedia is 'Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using on-hand data management tools or traditional data processing applications'. For me, that explanation nails it.
What it doesn't say is that big data can be when you have done something stupid and created a lot of data and don't have the skills to work with it. It could well be that what you perceive as 'Big Data' is simply you don't have the accessible skills to hand. Just because you have the tools, doesn't mean you have the skills to use them.
I think it's fantastic that businesses are now talking a lot more about data. There is absolutely interesting stuff out there and yes there are some clever new tools to allow you to grapple with all of this, but most companies very likely don't have this problem or quantity of data. They likely have a lack of the skill sets within the business to get a grip on things (especially early on). If you are working at a company and genuinely have such a quantity and complexity then great, but not everyone will. Just because there is a desire to have a new approach on your CV doesn't mean you should ignore what you have and shun a more simplified approach. Your employer is paying you for the right answer for them, not what's best for you.
Some places will genuinely need things like complex event processing and so on. Some places might genuinely generate sensor data at such a rate you can't work with it using traditional methods. That's great, embrace it and share what you find as we can all learn a lot from it. We shouldn't however, let a desire to use these tools force a more complex solution in than was needed.
The really good data guys I know simply treat the vast majority of all this as 'Business As Usual'. We've had a lot of data for some time and there are plenty out there with the skills to work with it properly (and many many more who don't). Those guys look beyond the end of their noses and can see where you are going with your data and how this giant puzzle all fits together. If you find them moaning it's because they've probably seen something that you don't have as a task yet, but if you did, you might well change your mind on how you are approaching the problem. You can solve the first stage problem today but if you have to replace it tomorrow it is likely wasteful. Yes, sometimes you need to do this, but forcing this in badly can only lead to tears. It is simply a case of looking at the big picture.
If you miss the basics of a problem early on you can end up generating what you perceive as Big Data. If you don't model the data appropriately you get data bloat, but that isn't Big Data. Big Data has a value that differs from the more traditional data but it takes a lot of 'different' effort to get that value out.
Let's take a little step back for a minute and look at the value vs the effort.
I quite often watch programs on Gold Mining on the history channel. What you see is a huge amount of effort for something really small and valuable. If you went purely off weight you are going to process thousands of tons for a couple of ounces. Those couple of ounces are financially well worthwhile having and absolutely must pay for the effort to get there. The quantity of earth moved to get there is huge though and it's easy to go bankrupt.
However, don't assume that all of the dirt is gold and once you've got the gold out, it's only worthwhile keeping that dirt if you then decide to look in it for diamonds.
Big Data is very similar to gold mining in many ways. Many people are affected by the Gold Rush. Plenty fail. There is definitely gold in them there hills, but if you don't have the right equiment and team at the right time then you aren't going to be the one to make their fortune.
No comments:
Post a Comment