Forget any arguments on the shape of the data you want to see (in and out). Forget which particular product is the preference and the maturity etc of that product.
It's some basic maths that sticks in my mind somewhat if you do everything as an 'in-memory' solution.
So....lets start with the basic maths.
I currently have a 1TB footprint of my data. I know I'm going to grow and if I give up a relational approach I'm likely to need some more space. Lets call it 2TB for the sake of argument. Hardly a massive data footprint as VLDB's are currently defined as starting at 1TB and data is growing a lot these days.
So, 2TB. In memory. That's a lot of memory.
I'm going to need 18 servers (with 60GB of RAM) per Terabyte. You can go for more memory if you like but you aren't helping your 'scale out not up argument'. It's an arbitrary number I suppose. I'll also suggest say £3k per server is likely.
I've solved having a solution like A-L and L-R and R-Z with my data split across nodes but I now have 18 single points of failure with data sharded across it.
Hmmm....Best split the data so that it never resides on just one node (so lets say 2 nodes is acceptable). That means that I need twice the memory footprint I was originally thinking of.
I'm now looking at 18 servers per TB which gives me 36 servers and now I need x2 of that to give me redundancy. 72 servers so far. Keep with me.
OK, I'm going to throw some more things into the mix. Yep, a second data centre for DR. Bugger. Erm....hang on a minute. 144 servers now.
I might be able to split the estate over two data centres so I need to have a think about the best balance for the business there. For now, let's stick with the 144 servers (yes it helps my argument, but it is my argument so ner).
Right...we have everything we need now don't we ?
Erm....No
- Dev
- Test
- QA
- Sandbox
Ah.....
Right, so I can bin some of the redundancy and keep them simple.
Lets say 144 servers for Live, 36 for Dev, Test, QA and Sandbox each = 288 servers.
288 x £3,000 = £864,000
Awesome. We now have everything we need !
No
Did you think about support ? Come on, you are looking at putting this live and you really should think about it. If you are talking MongoDB you are probably looking at £5k per server - call it £3k if you can negotiate a good deal.
That's another 144 x £3,000 which = £432,000
Ah
So we need 288 servers and have spent £1,296,000
And NOW we now have everything we need !
No !
Seriously ?
Have you thought about monitoring ? BI ? Reporting ? and so on ?
Righty ho. Bet you didn't think about power, cooling and rack space did you ?
Or switches to make the 288 servers talk to each other.
Or cables ? (OK, I'm being somewhat pedantic).
Lets say you use 300 watts per server. That is 300 x 288 = 86,400 watts ! (360 Amps)
I'm having a Doc Brown moment here if you can't tell.
At 42u per rack I'm going to need about 8 racks with some wriggle room. Assuming I can cool a rack full of pizza boxes with no gaps. Did you calculate the BTU's per hour that you need to cool ? No, I bet you didn't.
In case you haven't been to a data centre, that's a lot of floor space for 2TB.
What happens when you grow bigger than anticipated ?
Maybe, just maybe, you should pick your battles and work out what does need to be in memory (and realise that some solutions you already use are already using data in memory) and if you do genuinely need a very specific solution then go for it with the bits that will need it as opposed to everything at once.
For smaller data stores it does make sense, but there is a tipping point where you start to feel uncomfortable.
Maybe they only really make sense if they are cloud based and the hardware becomes someone else's problem. It does add other things to think about but I've leave it there for now.
Microservers or Blade Servers might be worth considering in order to reduce the physical footprint and possible cost (although you still have a lot to pay for)
