Building a Stable and Clustered J2EE Environment?

Slashdork posted the question (with a better than average data/noise response) and this reply which can be summed up as : Apache (with mod_jk2) load-balancing solution in front for the static content, JBoss + Tomcat on Linux clustered using JBoss' JavaGroups based clustering (buy the JBoss docs to help with the clustering), and the DB clustered behind that. More info here on Clustering with JBoss 3.0 from the Chief architect of JbossGroup.

On a related noted Jeremy Zawodny's recent blog entry High Availability is NOT Cheap does a good job of summing up why it's not cheap, and what you really need to be thinking about.

My own gut feel has always been : to get from 99% uptime to 3 9's (99.9) multiple the cost by 10 and from 3 to 4 multiple by 10, and from 4 to 5 multiple by 10. So you can likely get 99% with say $20K (2 boxes plus a few other things), 99.9 with $200K and 99.99 with $2m. If you haven't done that then your are cheapening out somewhere which will blow up in your face. also the other side of the equation is how many 9's do you need? 99.999% uptime is 5 minutes, 99.99% is 52 minutes, 99.9% is 9 hours, 99% is 9 days How much perk revenue and/or transactions does that site handle, worst/best case? How much do you lose if/when is down? Yes, there are also perception and risk issues. Clearly it makes sense for an Amazon.com to spend to get 5 or 6 nines, but just as clearly most organizations do not need more than 4 9's, and would do OK with 3 9's. Good management, operational and IT practices can add a nine, such things as : developing your existing staff; external and internal monitoring for potential problems; testing before, during and after development.

UPDATED: I wasn't thinking of salary costs when I went out on a limb and quoted some WAG dollar figures. Rule of thumb : Salary/development costs match hardware/software costs. If not then you missed something (or our trying to sell me something!). You might get away with spending 100K over 2 years (stretching it out) rather than 200k in one year, but if you try to get away with only 100k the first year your successor will just end of spending 200k fixing up the mess you left behind. Dev costs = HW/SW costs is another unbreakable iron triangle. Staffing numbers seem to have an optimum level: too little and you spending all your time task switching (although you can deliver on a longer time frame); too many and you spend all your time meeting/communicating (or the people who "do" are too removed from the people who "decide"). This is one of the management best practices they don't teach in the MBA schools. (Based on the number of times I been ask to metaphorical told to delivery the baby in less than 9 Months : it takes one man and woman 9 months, doubling staff count wouldn't change that, anything less than 9 months is less than a prefect baby and all the management directives to the contrary wouldn't change that. Sorry I had to get that off my chest. bad experiences. If any of this is new to you read : The Mythical Man-Month: Essays on Software Engineering by Frederick P. Brooks, ISBN: 0201835959 , and if it isn't new then we should re-read it anyway.)

Of course the cost (HW/SW) of doing all this is a tenth of what it was in 1999, and is likely to be at tenth of todays cost in 2008. Or to put it another way you can afford another 9 more than you could in 99. Using Linux/Apache/Jboss/MySql alone could save $100K to $200K in software costs ( and MySQL has transactions) over an AIX/Solaris Websphere/WebLogic DB2/Oracle build. and this is before the SAP/MySQL Deal which will yield results in bringing down the SW costs for ERP/CRM/BI and data warehousing in a couple of years.