We've struggled with this for years, these are some of the problems we've had over the years, hopefully can save some others the heartache we've had.
1) - Our Peak Season Setup for past 5 years
1 x MySQL Server / Load Balancer (Round Robin)
6 x Web Server
Web Server 1 was the master server and used rsync to updates files changes across the board,
â€¢ The system architecture of doing this method is severely bottle necked, we added more web servers to reduce load, but we were putting plasters on cuts rather than stopping severe bleeding.
â€¢ Storing all data / theme cache in "file" method is the most effective, but each web server wrote its own version to its own local var/cache folder, this caused major issues with the website looking for cache / variables that didn't exist. Mainly the issue for us here was product options not being viable to select / checkout. To solve this we mounted "var" as a network mount, but this created its own bottle neck.
â€¢ Thumbnail cache is dynamic like the website / theme cache, thumbnails created on server 1, was not present on server 2, rsync helped sort this, but we required 6 way read / writes to try sort this. In the end it was just far too complex. We created a network mount for "images", but again it created another bottle neck.
Our New Setup
1) Amazon RDS Database Server - This can be easily scaled / managed as required with no downtime.
2) Amazon EC2 Webserver - With careful planning this can be scaled up / down as required with minimum downtime.
We've upgraded to the 4.4.2 branch of the cart to take full advantage of the heavy architectural changes made in the cart. PHP7 is now fully supported too. We've also used Varnish and the Full Page Cache addon.
Our new website is much more simple, and performance is incredibly fast.
From performance point of view, sessions / cache etc should all use the "file" method.
We've still some stray mysql queries that need tweaking, and still had some EC2 issues that needed some tuning,