I've worked on a lot of systems since I left university in 1996. I've designed them, built them, supported them, re-worked them, tuned the hell out of them and indeed have broken some along the way.
With large scale high concurrency systems that have zero downtime you have to get your skills up. If you make a mistake you can quite often lose your employer a serious amount of money. For example, if you have 20,000 concurrent people on the website (not that big a number for a dotcom) you are going to find out very quickly if something isn't right. If you aren't informed then it's a problem.
The problem that we face is that a lot of companies look at their concurrency issues and immediately assume that they need Facebook or Twitter levels of solutions and therefore overcomplicate things.
Solve the problems you actually have and not the ones that you don't.
The trick to doing this is to keep things simple, monitor it, actually look at it and understand your monitoring and then (and this is the crazy bit) do something about it. You have to understand what you are asking of your systems. Obsess over the details if needs be but as a minimum communicate what you see and be passionate about it. People who don't understand you will often buy into the passion as they likely trust you know the technical detail. Ideally you should fail fast and fix fast. Don't tie yourself in knots adding more and more in. This is where Agile development helps a lot as you deal with smaller changes and can find problems easier.
A lot of companies will buy in monitoring systems and then put people in front of them who can follow process but don't actually understand what happened, why and what needs to be done to stop it happening again. They don't see what will happen when the problem happens again and then don't understand when it isn't worth fixing either.
I appreciate it's good to have the latest monitoring system for your CV, but you have to be able to understand it and react accordingly to benefit the business that employs you. You need a supportive structure to allow you to flag things. You need a business that will actively deal with technical debt as well as constantly pushing new features. You have to do both. If you don't you will cease being able to innovate and as your business adds more users (more high concurrency) you will stall at some point and likely lose your good staff.
From a development perspective, dealing with high concurrency changes the game somewhat and if I'm honest, a lot of developers just don't get it (the best ones do). In terms of developing a brilliant bit of software that has suitable tests wrapped around it for functionality, yeah, you can get a lot of developers without too much of a problem. TDD (Test Driven Development) doesn't often include testing things with a lot of users doing stuff at the same time. Tests should include load testing as a minimum with realistic load. Don't just fake out things from part of the stack, run the tests properly and re-create the problems you had in live recently. Then monitor, understand and solve them.
To do this properly you need the time to properly flush things out and get a routine going. Once done, it shouldn't be difficult to build in so that you have this automated. You need supportive people around you who let you address these things. If you don't, you end up being part of a system that is built to handle all of the compromises and not what was originally intended for the business or what the business expects it to do.
Take this to the extremes and the business will then bring in some highly paid people who think the problem is bigger than it is (as they have no idea of the detail that actually needs fixing) and then will try to apply bigger solutions as they think it is what is needed. As time progresses you find you have some more systems that don't really add anything and the underlying problems haven't gone away.
Treat every error as lost revenue. Go and count up all of your errors and put a nominal value against them. Scary isn't it ? When errors happen, think about that as lost opportunity and revenue. Always keep your live estate at the forefront as that's where the money comes from.
Monitor it - if you don't then how the hell do you know what's going on ?
Look at it - Seriously, go and have a look. This is where a lot of companies fail. See what happens when user connections go up, see what happens when IO maxes out or a component system becomes unavailable.
Understand it - Sounds easy. It isn't. What are the implications of a metric changing ? What else changes and why ? Don't just look at CPU, look at IO, look at processor queue lengths, look at memory use and indeed not just the quantity in memory, but the churn moving data in and out of memory. Look at the network packets and see if the data flowing out corresponds with the CPU and IO peaks. Delve in deep and see what's actually being done. Look at when metrics go down as well. If this isn't what you expected, is something wrong ? Is it right and we can do more of the same ?
Do something - What is the point of doing any of this is you can't effect some change to improve it ? Go and find the person who can change it and convince them. If that doesn't work, go up the ladder until you find the right person. If you keep running into obstacles, collate the issues together, don't give up yet. There is a tipping point whereby you walk away but often this can be avoided.
No comments:
Post a Comment