Switching from Squid to Varnish (and getting some nice benchmarking tools along the way)

We recently (November 19th) switched from Squid to Varnish for serving requests on cnx.org. The new configuration improved performance for content visitors and gave us a chance to test some new performance benchmarking tools.

We are continuing to get cnx.org ready for a major overhaul of caching to improve performance and to make sure that the tools that we use have a broad base of developer support. Some of our development and consulting partners have more experience with Varnish, and so we decided to take a look at Varnish as a potential replacement for Squid. (An earlier blog post discusses our switch to HAProxy for load balancing).

Getting repeatable tests using a cnx.org workload was the first step in our series of performance milestones. We hired Enfold Systems to develop a set of FunkLoad tests using common URLs from Connexions. The benchmark hardware mimics the cnx.org configuration and the benchmarks increase the number of concurrent requests using a set of URL's from our cnx.org logs.

This trac page has more details about our FunkLoad tests and will have links to the reports from various benchmarks as we run them.

One representative graph from the benchmarks is shown at the top of this blog. Varnish results are on top (Y axis 12/sec) and the Squid results(Y axis scale 6/sec) are below. The graphs show the number of pages returned per second as the concurrent visitors are cranked from 10 to 50. Our Varnish configuration serves about double the number of pages and doesn't experience the degradation that our Squid configuration experiences at 30 concurrent visitors. The caveat, of course, is that these results only apply for the particular benchmark and configurations that we set up for each.

The results of actual performance on cnx.org when we switched from Squid to Varnish show an improvement in the time to serve non-authenticated content (almost all of our traffic). The benefits do not appear to be helpful for authors, however. We are upgrading the benchmarks to include authenticated URLs as well so we can test performance for both. In the graph below, the Squid to Varnish change occurred at the end of week 46. The Y axis shows the time to serve a particular URL. Connexions clearly still suffers from spikes of long page delivery times, but the overall averages and the minimum serve times came down.

Comments and suggestions are welcome.