Having a detailed look into things made me think a bit more about the structure of data within a JSON/BSON document.
An example
{
"CN" : "Something here to denote the name of the customer",
"AB" : 27,
"_id" : {
"ST" : "MANCHESTER",
"TS" : ISODate("2014-02-05T15:00:00.000Z"),
"CI" : 158580,
"XX" : 0
}
}
Let's do the architect thing here and colour it in.
{
"CN" : "Something here to denote the name of the customer",
"AB" : 27,
"_id" : {
"ST" : "MANCHESTER",
"TS" : ISODate("2014-02-05T15:00:00.000Z"),
"CI" : 158580,
"XX" : 0
}
}
- The stuff in Yellow is your structural meta data for this document.
- The stuff in Red is data that is highly likely repeated in similar documents
- The stuff in Green is the useful data.
If I drop this into a Word or Google Doc and do a quick count I can see I have 190 characters with spaces and 152 without. The document breaks down as follows :
- Spaces = 38 characters (20%)
- Meta Data = 61 characters (32%)
- Repeatable Data = 59 characters (31%)
- Useful Data = 32 characters (17%)
If you modelled this with an RDBMS you could get this down to approx 20 characters.
One of the biggest arguments FOR a NOSQL option is the ability to be schema free as such. Well, if you are to go schema free then that's fine. If however, you are going to repeat that exact same structure in many millions of documents then it misses the point and you might as well have a more fixed structure.
I was chatting with a friend about this (he's a very good Dev btw) and he asked 'well, how does SQL handle this meta data ?'. It's a good question and I guess that most none data guys haven't really thought about it.
The answer is simple. It gets stored once. That's it. Once. We pay a storage cost that is the same if you have 1 row or 1 million rows.
With JSON, you store it in each and every single document.
Let's play that out a little and put some costs against it.....
190 characters (or bytes if you prefer) and really only 32 useful ones. Let's say we create 20 million of these documents a day and keep a rolling 365 days.
That's 3.8 Billion Bytes in a day. Just over 3.5GB per day. Over a year that's going to be 1.2TB or thereabouts.
Now let's put a cost against that. Let's say £4K/year/TB so we get £4.8K in costs.
If you didn't have all of the extra 'stuff' that would have cost you £816.
Oops.
Lets say we have some (4 core) servers with 128GB of RAM and they cost £3K each for the hardware and £3K/year for hosting with a premium company. Let's just assume we need the same amount in memory and on disk so we don't lose it.
With the extra stuff we need 1,200TB / 128 * 2 (basic redundancy) = 19 servers (call it 20 for ease of use). I need £60K for hardware and £60K/year to run them.
My 3 year total cost is then £255K.
If I did the same with the useful data it works out as (1,200 / 100 * 17) = 204GB of data. So, 204 / 128 * 2 = 3.2 servers (lets call it 4). Cost wise that's £12K of servers and £12K/year of hosting + £816/year of storage. That's just under £25K all in.
£255K vs £51K for 3 years.
Half a rack of servers vs 10% of a rack.
Half a rack of servers vs 10% of a rack.
Bloody Nora ! Sobering isn't it ?
Your Big Data problem is in the meta data needed to define the structure. You can't glean anything useful from the braces and colons used to define the structure.
Yes, but NOSQL stores are fast !
Erm....yes, you have just gone from 16 cores to 80 cores so it better had be.
Good job you ain't paying for them expensive SQL licences eh ? Erm...you are looking at about £5K/server/year for Mongo support so you've spent another £100K/year on them that we didn't account for.
SQL wise, you are looking at $7K per core. So 16 cores = £67K for perpetual licences (and that's for Enterprise edition which performs way better than the Standard version people normally compare to). We don't have to pay for failover so we can halve that amount too.
So far we have a 3 year cost as follows :
JSON option = £555,000
None JSON option = £89,000
Difference = £466,000 cheaper by handling JUST the useful data.
None of this covers indexes and the space needed for them.
So....in conclusion, what you perceive as a Big Data problem might actually not be. IF you are going to go schema free then use it where a fixed schema really isn't the right fit. To be honest, if you are creating 20m documents a day and they all differ completely in structure then you do likely have other problems, but if they are the same then take a proper look at the data to see if you can work with it better.
Do you want to be the guy who asks for £466,000 that isn't going to give the business any value whatsoever ?
Do you want to be the guy who asks for £466,000 that isn't going to give the business any value whatsoever ?
Its an interesting comparison but, in my opinion, the big data method should not be used to store 3 years of data. Its the wrong application. Sure people are doing it but they are wrong.
ReplyDeleteBig data should be about flexibility and the ability to scale out rather than up and not much else. RDBMS should be about scale up and long term storage where schema is more rigid.
I think the two technologies can compliment each other but neither can replace the other.
Dave
Dave,
ReplyDeleteI agree.
In this particular space that we are looking at (analytics of real time data) I'm looking more at something like StreamInsight or Storm for Complex Event Processing which initially looks a more suitable fit. What I was hoping to achieve with the post above was to show that the meta data can give you a different set of problems to those people might expect to have.
Ryan
Ignore the spaces, they only need to exist for human readability and not for storage.
ReplyDeletebut....if they are included and the file not compressed then they still count. Currently looking at CouchBase (meeting them on Mon) and have taken a short term decision to use Google Big Query. CouchBase apparently can compress the data so that makes a big difference. GBQ looks pretty decent for what I need it for and I don't have to worry about a lot of the other problems I would with other options. Will see how we get on.
ReplyDeleteGBQ is there is a longer term persistence and not needed for anything real time or OLTP type time frames.
ReplyDelete