Big Data or JSON Meta Data Problem ?

I've been looking at a few things with MongoDB recently with some of the guys at work. It was brought to me as a 'Big Data Problem', but I think the problem lies in the meta data.

Having a detailed look into things made me think a bit more about the structure of data within a JSON/BSON document.

An example

{

    "CN" : "Something here to denote the name of the customer",

    "AB" : 27,

    "_id" : {

        "ST" : "MANCHESTER",

        "TS" : ISODate("2014-02-05T15:00:00.000Z"),

        "CI" : 158580,

        "XX" : 0

    }

}

Let's do the architect thing here and colour it in.

{

    "CN" : "Something here to denote the name of the customer",

    "AB" : 27,

    "_id" : {

        "ST" : "MANCHESTER",

        "TS" : ISODate("2014-02-05T15:00:00.000Z"),

        "CI" : 158580,

        "XX" : 0

    }

}

• The stuff in Yellow is your structural meta data for this document.
• The stuff in Red is data that is highly likely repeated in similar documents
• The stuff in Green is the useful data.

If I drop this into a Word or Google Doc and do a quick count I can see I have 190 characters with spaces and 152 without. The document breaks down as follows :

• Spaces = 38 characters (20%)
• Meta Data = 61 characters (32%)
• Repeatable Data = 59 characters (31%)
• Useful Data = 32 characters (17%)

If you modelled this with an RDBMS you could get this down to approx 20 characters.

One of the biggest arguments FOR a NOSQL option is the ability to be schema free as such. Well, if you are to go schema free then that's fine. If however, you are going to repeat that exact same structure in many millions of documents then it misses the point and you might as well have a more fixed structure.

I was chatting with a friend about this (he's a very good Dev btw) and he asked 'well, how does SQL handle this meta data ?'. It's a good question and I guess that most none data guys haven't really thought about it.

The answer is simple. It gets stored once. That's it. Once. We pay a storage cost that is the same if you have 1 row or 1 million rows.

With JSON, you store it in each and every single document. 

Let's play that out a little and put some costs against it.....

190 characters (or bytes if you prefer) and really only 32 useful ones. Let's say we create 20 million of these documents a day and keep a rolling 365 days.

That's 3.8 Billion Bytes in a day. Just over 3.5GB per day. Over a year that's going to be 1.2TB or thereabouts.

Now let's put a cost against that. Let's say £4K/year/TB so we get £4.8K in costs.

If you didn't have all of the extra 'stuff' that would have cost you £816.

Oops.

Lets say we have some (4 core) servers with 128GB of RAM and they cost £3K each for the hardware and £3K/year for hosting with a premium company. Let's just assume we need the same amount in memory and on disk so we don't lose it.

With the extra stuff we need 1,200TB / 128 * 2 (basic redundancy) = 19 servers (call it 20 for ease of use). I need £60K for hardware and £60K/year to run them.

My 3 year total cost is then £255K.

If I did the same with the useful data it works out as (1,200 / 100 * 17) = 204GB of data. So, 204 / 128 * 2 = 3.2 servers (lets call it 4). Cost wise that's £12K of servers and £12K/year of hosting + £816/year of storage. That's just under £25K all in.

£255K vs £51K for 3 years.

Half a rack of servers vs 10% of a rack.

Bloody Nora ! Sobering isn't it ?

Your Big Data problem is in the meta data needed to define the structure. You can't glean anything useful from the braces and colons used to define the structure.

Yes, but NOSQL stores are fast !

Erm....yes, you have just gone from 16 cores to 80 cores so it better had be.

Good job you ain't paying for them expensive SQL licences eh ? Erm...you are looking at about £5K/server/year for Mongo support so you've spent another £100K/year on them that we didn't account for. 

SQL wise, you are looking at $7K per core. So 16 cores = £67K for perpetual licences (and that's for Enterprise edition which performs way better than the Standard version people normally compare to). We don't have to pay for failover so we can halve that amount too.

So far we have a 3 year cost as follows :

JSON option = £555,000

None JSON option = £89,000

Difference = £466,000 cheaper by handling JUST the useful data.

None of this covers indexes and the space needed for them.

So....in conclusion, what you perceive as a Big Data problem might actually not be. IF you are going to go schema free then use it where a fixed schema really isn't the right fit. To be honest, if you are creating 20m documents a day and they all differ completely in structure then you do likely have other problems, but if they are the same then take a proper look at the data to see if you can work with it better.

Do you want to be the guy who asks for £466,000 that isn't going to give the business any value whatsoever ?

How to impress the DB guys with NoSQL

Lets step away from the technical aspect of SQL vs NoSQL for a minute. 

We know there are some core differences between an RDBMS approach and a document store approach (or KVP, Graph or Column Store). There is a way to get the more traditional data guys on-board.

This is totally one-sided by the way. There will be an equal argument from various Devs and if a response is well written, I promise I will add it. My hope is to provoke some thought.

Attitude

First up, don't seperate yourselves and then claim to be better. That's just never going to win any friends. OK OK so you don't like the DB guys and don't want them as friends, but hey, lets all play nicely. It might well be better or be a better fit but lets go over the basics first without looking purely at headline figures.

Don't forget that when something goes wrong code wise and calls the DB a few thousand times a second when it shouldn't, it's the DB guy who gets woken up at 3am to have a look. Even if you know it's not a DB issue, that guy still gets called for something he probably wasn't involved in. No wonder he gets pissed off and is grumpy and wants to protect his part of the estate. A little bit of being humble would go a long way on both sides. If the DB guy fucks up then the company stops earning and it's blatantly obvious. It's a lot of pressure. I've been there and it's tough. Rolling back DB changes needs a lot more thought. Building a system that won't lose data isn't easy.

Understanding the differences in approach

When coding, YOU tell the computer what to do and how to do it. With a DB you tell it WHAT you want and it figures that out for you. Don't forget that everyone else is competing for the same resource if your system is complicated. The DB engine will figure out things based on a least cost route. It's not always what you perceive to be best.

One of the biggest failings of RDBMS's is that they let you do stupid things and don't tell you if you asked in a crap way. It takes effort to get these things right. That's why things take time to develop. It takes more time to do DB development than pure coding. I've done both and it genuinely does. It pays off bigger if you get the data right too.

'Everyone can write a SELECT statement Ryan'. True, but not everyone can build a highly performing, highly available database. So let's not patronise each other or take cheap shots.

Backups

One of the fundamental things you do as a data professional is protect the data and make sure you can recover should something go wrong. It does go wrong and any long term data guy has been bitten. 

Do you want to be the guy sitting there without recoverable data saying 'erm...sorry' ? Hell no.

We genuinely care about what we are looking after. Any DB guy worth his salt will cover backups on his first hour of day one in a new job. Pretty much everything else can be worked on but without sorting this out then the rest is pointless.

Backups should be tested and proven. If you aren't doing this, you aren't doing it right. It should not be an afterthought. It gets interesting when you look at NoSQL solutions where you have to take a copy of the physical data from under the data store. How about across shards ? Can you get a consistent copy across shards without data loss and in a consistent state ? Not quite that easy is it ? OK so there is eventual consistency but is it really just consistently inconsistent ? It's worth thinking about how you do this right.

Performance

Yes, NoSQL solutions can be fast. However, in order to be fast, get the people who can make it fast. Don't just assume that out of the box it will be fast. The DB guys know a lot about making data go fast. They understand how to structure a b-tree to make it a simple pass instead of overcomplicating things (yes, NoSQL solutions such as MongoDB use b-tree's so a lot can be learnt). If you are going to choose a NoSQL solution then get the guy that knows how to make that work brilliantly. I can learn a lot from that guy.

I totally get that these solutions can be quick, but seriously here. Prove it. But work with me as we prove it together. When you genuinely blitz what we do you will raise eyebrows I promise. You will have our attention. Sure you can do something fast with MongoDB, Vertica or whatever but if you claim it's going to be fast, then make it properly quick. Do that under serious load with a lot of data and you will have my attention I promise. How fast is fast enough ?

There is no point in you buying a motorcycle if you don't know how to ride it.

Know that you aren't Amazon or Facebook

They have a whole bigger and scarier problem. You don't need their approach any more than I need a fleet of helicopters to travel to work. You can learn a lot from them but I bet you don't have their problems, resources or turnover.

Think of the bigger picture

If you come along with an interesting approach that covers a lot of the basic bases the DB guy will be impressed and helpful. Don't just look at one answer. The fastest way to the front door at work is to jump out of the window. It might not be the best approach though. If you apply the same thinking as the DB guys might then you might save yourself some pain.

The platform

For me I try to architect and build the platform for the biggest component piece that I have in front of me. Once that's done I will move on to the next one until it's right. For my current work it's a very busy OLTP approach and we have paid for MS SQL Enterprise. It would be irresponsible to not use that to it's fullest. Where it doesn't fit we will find the right approach, but if it works for 80% then I should spend 80% of my time there.

Understand we don't always suggest SQL

I get accused of this a LOT. I know it extremely well yet still have a long way to go. I fully appreciate that there are failings in SQL, don't overly like MySQL and think Oracle is too expensive. I know it can be seemingly overly complex and expensive but it is a superb product with a lot of time spent understanding it and making it better. I guess it's more of a reluctance based on seeing problems in other areas and I'm still looking for the one that totally blows my socks off. I get handed MongoDB as the answer a lot without being given the question it is supposed to answer. I genuinely like Neo4J. ElasticSearch (with Kibana) looks great where I know the problems we wanted to solve and it is the right fit. I'm all for it when it's right.

Whatever answer fits best should be based on the right answers for you. One size doesn't fit all, but the arguments distract from finding the right answers.

Work with us

I dealt with a Chief Architect several years ago who said 'stick around and you might learn something'. To me that summed up his arrogance. For me I would rather say 'work with me, learn from me and teach me'.