I've been thinking about NoSQL solutions a lot recently. It comes with the territory I guess.
If you imagine any radical change as a swing, we've probably come to the extreme at this stage and are swinging back the other way somewhat. That is back towards RDBMS's. I'd imagine that over time things will settle part way between the two with the sucessful vendors being the ones that offer the most features of both. NoSQL options are adding in SQL interfaces and discovering the problems that RDBMS's have gone on to address over the years (indeed they are likely to make similar mistakes).
Google have certainly found that NoSQL solutions and the ideas of Eventual Consistency have caused the sufficient problems to move back to an RDBMS (but retaining the nice features that NoSQL offer)
http://www.theregister.co.uk/2013/08/30/google_f1_deepdive/
Google also use RSBMS's heavily (having recently decided to move to MariaDB from MySQL).
I started to dig into the NoSQL thing a bit more and from what I can see it mostly seems to stem back to a white paper from Amazon about scaling data at extreme scale. If I can find the link again I'll update this blog post.
The basic premise of the paper highlighted something that Ted Codd also pointed out back in the 1970's as he worked on the foundations for relational databases. This was that there is an overhead in checking constraints on data in a relational system. So think of there being overhead for foreign keys and you get the idea. It is absolutely true in that there has to be some delay while that work is done (we are talking very small fractions of time though depending on scale). Amazon's paper found that at 'extreme scale' this caused a problem for them with the expectations they had for their systems.
Part of the problem that Amazon was aiming to solve was that they needed systems that would guarantee writes for the 99th percentile of their customers. In most cases you would typically look at the 95th percentile, so 99th is a whole different argument as it brings in most of the edge cases and that can change the game.
Do this at 'extreme scale' (with very high concurrency) and it becomes a very difficult problem to resolve.
Amazon also aimed to address the 'delays' that you get from an RDBMS which is in the locking, logging and latching of the data which most developers perceive isn't fast enough for their needs. The best approach I've seen to improve on this is from VoltDB (another Michael Stonebraker product). It's early days for this and while they only offer support in office hours EST it will stay off my radar. They are both absolutely right in that there is time lost here. However, how much time and what impact that has for you is an interesting question. If we are talking the last few microseconds then yes it is valid, but if that doesn't really matter to you do you need to worry ? If you have delays introduced by poor coding etc... then you likely aren't at the stage where you need to worry here.
The NoSQL options here stop you worrying as it's generally a fire and forget approach. This is fine until something goes wrong. To help make it better (with say MongoDB) you can turn on journalling, but then you have re-introduced a basic concept you were trying to avoid. Hmmm...
Amazon were quite right in that the delays from a relational system would likley impact their expectations so they created their own approach which is where the NoSQL approaches stem from. It might be possible to argue that you could overcome a lot of the problems that both Amazon and Codd identified, but I'm sure they have all thought it through so lets give them credit and assume they were right.
We now move onto the 'joins are bad' argument. Feel free to read my other post on this, but as a basic statement I'd say that people misunderstanding how to work with data is the problem and that joins are perfectly fine for the vast majority. Those arguing against them are likely developers and I'd suggest they probably don't know RDBMS's enough to have a qualified discussion.
A lot of latency in databases comes from storage whichever option you choose. Storage desperately needed to move on and has done so now with NAND flash (even though it is a 25 year old idea). Moores law doesn't apply to spinning rust so it's hampered data as we've exploded the use of it.
What we need to remember is that Amazon was aiming to solve Amazon's problems. Not yours. They have offered DynamoDB to the public for a while but that's sensible to make revenue from it once they had implemented it for themselves first.
A number of other approaches have come along the way and offer solutions that allow you to move away from RDBMS's. Do you really need to solve the problems that Amazon needed to ?
Most of the NoSQL options appear to try to solve the problems that they understand from Amazon's findings. Most businesses don't need to scale to the extremes that Amazon is at. Graph databases might not be new, but Neo4J looks very interesting and I see good potential there in the right places. MongoDB is based on ideas from Lotus Notes which was awful IMHO (I spent 6 months as a Lotus Notes developer back in the day).
In terms of which platform to choose ? Well, it's an entirely different question if you are a start up or have an established platform. The true TCO can be way higher than expected if you swapped it all in (especially if you are looking at this for an Enterprise scale business). The costs look good for single servers, but try and see how much it costs for trying to put 1TB into RAM in a sharded setup. It ain't as cheap as you might think, especially when you consider support and redundancy or failover, dev, test, qa and so on. You can achieve this with lower costs if you trade off RAM for fast enough storage (say FusionIO) but you loose the scale out argument. Each approach is a balance of trade offs. Yes MS charge a lot and Oracle more. It's a stick I get beaten with most weeks and I wish it was better.
Developers often think that saving costs and time for development is the only consideration. It is very important yes, but if I save a few grand on Dev time but spend £3m in production then it's not an easy sell to me. It's simply because of different needs of the data that stops each side seeing the other's argument. (For the record, I've been a Dev for many years, then SDM, then DBA, then Data Architect and now Enterprise Data Architect who also owns a small software business).
The Agile movement has been incredibly worthwhile and beneficial but we might have gone a bit too far. In the lean world you only think about the next problem for development. However, from an enterprise architecture point of view I'm interested in making sure that when you come to the bigger problems we haven't taken the wrong direction earlier on. I'd love to hear a developer or SDM show me where the maths makes sense for the business whereby the changes we make are going to bring in a higher return and cover it's basic costs. It's the most basic concept of business and yet is often forgotten in the rush to develop the next feature. It'll get you somewhere down the road if you are lucky (and half decent) but at some point you have to turn that into profit.
Start with the basic maths. How big is the data. What will it take to run it. How fast do you need it. Can you afford to change it and if so which gives you the best return ? Don't just start coding. Measure twice, cut once my Dad always used to say.
So...right tool for the job yes (polyglot persistence), but a bit of common sense and some basic maths would go a long way on both sides. Having a platform that does 90% and then using something else for the edge cases is far more sensible. Remember that NoSQL stands for 'Not Only SQL' instead of 'Anything But SQL'.
