Switching from Squid to Varnish (and getting some nice benchmarking tools along the way)

We recently (November 19th) switched from Squid to Varnish for serving requests on cnx.org. The new configuration improved performance for content visitors and gave us a chance to test some new performance benchmarking tools.

We are continuing to get cnx.org ready for a major overhaul of caching to improve performance and to make sure that the tools that we use have a broad base of developer support. Some of our development and consulting partners have more experience with Varnish, and so we decided to take a look at Varnish as a potential replacement for Squid. (An earlier blog post discusses our switch to HAProxy for load balancing).

Getting repeatable tests using a cnx.org workload was the first step in our series of performance milestones. We hired Enfold Systems to develop a set of FunkLoad tests using common URLs from Connexions. The benchmark hardware mimics the cnx.org configuration and the benchmarks increase the number of concurrent requests using a set of URL's from our cnx.org logs.

This trac page has more details about our FunkLoad tests and will have links to the reports from various benchmarks as we run them.

One representative graph from the benchmarks is shown at the top of this blog. Varnish results are on top (Y axis 12/sec) and the Squid results(Y axis scale 6/sec) are below. The graphs show the number of pages returned per second as the concurrent visitors are cranked from 10 to 50. Our Varnish configuration serves about double the number of pages and doesn't experience the degradation that our Squid configuration experiences at 30 concurrent visitors. The caveat, of course, is that these results only apply for the particular benchmark and configurations that we set up for each.

The results of actual performance on cnx.org when we switched from Squid to Varnish show an improvement in the time to serve non-authenticated content (almost all of our traffic). The benefits do not appear to be helpful for authors, however. We are upgrading the benchmarks to include authenticated URLs as well so we can test performance for both. In the graph below, the Squid to Varnish change occurred at the end of week 46. The Y axis shows the time to serve a particular URL. Connexions clearly still suffers from spikes of long page delivery times, but the overall averages and the minimum serve times came down.

Comments and suggestions are welcome.

HA_Proxy load balancing leads to better Connexions performance

We have two different performance milestones coming up at Connexions. One to improve performance of authoring by reducing the size of certain catalogs where lookup time is related to size, and one to improve viewing times by making our slowly changing content cacheable.

Getting ready for these performance milestones, we are updating our basic request architecture and we have made some nice performance improvements in the average times for viewing content and using the authoring system. And this is before starting on the real performance work.

We were using Squid to load balance between front end zeo servers and we switched to using HA_Proxy for load balancing. Take a look at these graphs of average load times. The Y-axis shows the time to service a particular request. We measure a few different requests and each shows up in a different color. The graph on the left shows performance for the last month and the one on the right shows performance for the year. You should notice a dramatic drop in service times a week and a half ago when HA_Proxy was added for load-balancing.

The graph below shows the same timings, but for actions that authors take. It shows a similar speedup (lower height lines) for authoring. In the graphs for authoring you may also notice a major performance improvement in February of this year. That improvement was thanks to increasing the size of an application object cache.

So why would changing the load balancer have such a big benefit? We were surprised at the magnitude, but not the direction of the improvement. Squid isn't specifically designed to do load balancing, and we were using the Internet Cache Protocol (ICP) to approximate load balancing. Squid would ask frontends to respond to an ICP request and then use the speed at which they responded "no" to determine which one to choose for the request. HA_Proxy is designed to do load balancing. It did take some configuration, but it is working much better than trying to contort ICP for load balancing. Our settings for HA_Proxy choose the front end with the least number of current connections and then choose among equals using round-robin.

As background about our characteristics, Connexions serves about 2 million unique visitors per month. We receive between 50 and 60 requests per second, peaking at around 100 requests per second. Our performance is still spiky. Occasionally a request takes a very long time to serve, and that can be very frustrating for viewers and authors. We will continue to report on performance in the blog as we improve the infrastructure.