See More
Popular Forum

MBA (4887) B.Tech (1769) Engineering (1486) Class 12 (1030) Study Abroad (1004) Computer Science and Engineering (988) Business Management Studies (865) BBA (846) Diploma (746) CAT (651) B.Com (648) B.Sc (643) JEE Mains (618) Mechanical Engineering (574) Exam (525) India (462) Career (452) All Time Q&A (439) Mass Communication (427) BCA (417) Science (384) Computers & IT (Non-Engg) (383) Medicine & Health Sciences (381) Hotel Management (373) Civil Engineering (353) MCA (349) Tuteehub Top Questions (348) Distance (340) Colleges in India (334)
See More

High scalability technology stack [closed]

General Tech Technology & Software

Max. 2000 characters



( 3 months ago )

I'm building a webservice that is going to be under ridiculous load (thousands to ten-thousands of queries per second). My normal stack of apache, PHP, memcache and some DB will be able to handle it with a nice load balancer infront and lots of machines, but I'm wondering if there are better solutions.

The endpoint will be hit by a beacon (via javascript on the client), I'll read the user's cookies, pull some small info on them from the DB, cache it, do some small calculation, send the response and if needed write to the DB and invalidate the cache.

And good technology choices and/or hardware recommendations?



( 3 months ago )

This isn't the kind of question that can be answered here in anything other than a broad overview. Some general pointers:

  • Hardware: the two choices are basically lots of small, cheap boxes or fewer number of more powerful boxes. Cheaper boxes are, well, cheaper but typically consume a lot more power for the same CPU or memory (whichever is important to you) than bigger boxes. People often forget about the sometimes significant cost of power consumption;
  • Backend: you have a few choices from the big end of town (Oracle, SQL Server) to the commoditzed end (MySQL). MySQL is obviously cheaper and you can go far on MySQL but there is no question that Oracle (which I'm more familiar with than SQL Server) has a better optimizer, is more capable and is more robust than MySQL. You will however pay for it;
  • Budget: this is a huge factor as it might be worth paying for good commercial software rather than paying development costs to use "free" software. Software development is one of the most expensive costs of all;
  • Vertical and horizontal scalability: the question you're basically seeking to answer here is do you build up (bigger boxes, etc) or build out (clustered environments). The most scalable solutions have near-linear horizontal scalability but in the shorter term vertical scalability can be cheaper.

As for your normal stack, I'd stick with it unless you've got a particular requirement you haven't mentioned that prohibits it. After all PHP is a proven technology that runs 4 or so of the top 20 sites on the Internet (Facebook, Wikipedia, Flickr and I think Yahoo). If it's good enough for them, it's good enough for you.

More importantly, you know it. Technology stacks you know trump technology stacks you don't in almost every case. Beware the "greener pasture" trap of the latest hyped-up technology stack.

Memcache is good. The other thing you might want to consider adding to the mix is beanstalkd as a distributed work queue processor.

One important question to answer is: how well can you partition your application? Applications that easily lend themselves to partitioning are far easier to scale. Those that aren't tend to be modified in some way to make them easier to partition.

A good example of this is a simple sharetrading application. You can could partition market information based on stock code (A-C on one server, D-F on another and so on). For many such application that will work well.

what's your interest