Connexions/Rhaptos Sprint February 10th, Houston TX

Calling all developers. Come join us after the Connexions Conference 2011 , for a coding sprint, on February 10th from 9 AM to 3 PM in Houston at Rice University. We will have a mix of expert Rhaptos/Connexions developers, Plone and Zope developers, and new developers, just getting started with Rhaptos. We will be continuing the Plone 4 upgrade sprint started in Bristol under the direction of Upfront Systems. For new developers, we will have some juicy bugs to tackle with the help of expert partners. And for both developers and designers, we will be working with some prototype editors for creating Connexions content. User experience practitioners are welcome also. In addition to the editors, we will have usability bugs to decipher and solve.

If you can't make it to Houston, but would like to participate remotely, we will have an IRC chat and a Skype voice session set up.

Please let us know by emailing cnx@cnx.org, if you would like to participate and we will send instructions late this week about a few things you can set up before the sprint to get ready.

Hope to see you Thursday after next!

Introducing J. Cameron Cooper

“I am pleased to introduce Consortium technology committee member, Cameron Cooper, independent consultant and Plone expert. Cameron knows Connexions software extremely well. He was Connexions System Architect for several years and we are extremely lucky to have his continued participation in the Connexions Consortium technical committee. His expertise was instrumental in the development of strategically important new Connexions features, including the collection composer, the print-on-demand system, and lenses, the post-publication, quality-review system critical in an open publishing repository. His combination of deep knowledge of Plone and Zope, and detailed knowledge of the ways in which Connexions adapts and extends them, makes Cameron a valuable long-term contributor to the Consortium and the Technology Committee. Below, Cameron discusses the big vision frontiers that he sees for Connexions and Rhaptos."
Kathi Fletcher -- Technology Director and Project Manager at Connexions

Hi, J. Cameron Cooper here (jccooper@jcameroncooper.com). I'm the former Systems Architect for Connexions, and still serve on the Connexions Consortium Technical Committee, where I hope my hard-earned knowledge will continue to be of use. I started with Connexions in 2003 to port the system from a plain Zope product to the current Plone architecture. I became System Architect in 2006 when Brent Hendricks, the previous (and excellent) technical lead left. Connexions has grown to a major web application with 10s of thousands of lines of code and over 2 million visitors per month. And I'm very proud to have been (and to continue to be, though less day-to-day) a part of that. I'm occasionally an independent web developer, writer, and trainer and I'm currently running my own company called BottleMark, making custom bottle caps—as well as doing all the web dev for the site.

While working at Connexions, I became a major Plone expert; my book came out in late 2004, and I spoke at Plone conferences in 2003, 2005, and 2006. And that's no accident: Connexions is an extremely ambitious use of Plone, and customizes essentially every subsystem.

As part of the technical committee, the priorities I see for Rhaptos and Connexions are:

1. ensure a scalable and maintainable codebase. Connexions has grown significantly both in usage and in complexity, and the system needs to be able to keep up.
2. transition to a distributed development model. Connexions has been developed by a core group, but now must operate more as an open source project. The committee is exploring various ways of funding development and working with external developers, and this is very important to growing the feature set and usefulness of the software.
3. distribute the content repository. Connexions remains a standalone system, but to support massive content and usage, I forsee a need for the system to live in multiple places.
   

I'm hoping to work for the committee on a pilot project to harness crowdfunding of development, where interested parties provide support through a website like Kickstarter. I am investigating small but useful projects that will take about one week of software development; perhaps "Enhanced Author Profiles". I am interested in other ideas that could be done in about a week—so let me know if you have one.

Connexions for Android Updated

The Connexions for Android app has been updated. The changes are detailed on the Connexions blog. You can download the app and view screenshots on the Connexions for Android splash page. It is also available in the Android Market and in AppsLib. The source code is available in our SVN Repository. Let us know what you think at techsupport@cnx.org.

Switching from Squid to Varnish (and getting some nice benchmarking tools along the way)

We recently (November 19th) switched from Squid to Varnish for serving requests on cnx.org. The new configuration improved performance for content visitors and gave us a chance to test some new performance benchmarking tools.

We are continuing to get cnx.org ready for a major overhaul of caching to improve performance and to make sure that the tools that we use have a broad base of developer support. Some of our development and consulting partners have more experience with Varnish, and so we decided to take a look at Varnish as a potential replacement for Squid. (An earlier blog post discusses our switch to HAProxy for load balancing).

Getting repeatable tests using a cnx.org workload was the first step in our series of performance milestones. We hired Enfold Systems to develop a set of FunkLoad tests using common URLs from Connexions. The benchmark hardware mimics the cnx.org configuration and the benchmarks increase the number of concurrent requests using a set of URL's from our cnx.org logs.

This trac page has more details about our FunkLoad tests and will have links to the reports from various benchmarks as we run them.

One representative graph from the benchmarks is shown at the top of this blog. Varnish results are on top (Y axis 12/sec) and the Squid results(Y axis scale 6/sec) are below. The graphs show the number of pages returned per second as the concurrent visitors are cranked from 10 to 50. Our Varnish configuration serves about double the number of pages and doesn't experience the degradation that our Squid configuration experiences at 30 concurrent visitors. The caveat, of course, is that these results only apply for the particular benchmark and configurations that we set up for each.

The results of actual performance on cnx.org when we switched from Squid to Varnish show an improvement in the time to serve non-authenticated content (almost all of our traffic). The benefits do not appear to be helpful for authors, however. We are upgrading the benchmarks to include authenticated URLs as well so we can test performance for both. In the graph below, the Squid to Varnish change occurred at the end of week 46. The Y axis shows the time to serve a particular URL. Connexions clearly still suffers from spikes of long page delivery times, but the overall averages and the minimum serve times came down.

Comments and suggestions are welcome.

Enterprise Rhaptos 1.6 Released

Enterprise Rhaptos 1.6 has been released. It contains the new Collection Composer recently added to cnx.org as well as a handful of bug fixes.

See the upgrade instructions and the list of updated products for more details.

HA_Proxy load balancing leads to better Connexions performance

We have two different performance milestones coming up at Connexions. One to improve performance of authoring by reducing the size of certain catalogs where lookup time is related to size, and one to improve viewing times by making our slowly changing content cacheable.

Getting ready for these performance milestones, we are updating our basic request architecture and we have made some nice performance improvements in the average times for viewing content and using the authoring system. And this is before starting on the real performance work.

We were using Squid to load balance between front end zeo servers and we switched to using HA_Proxy for load balancing. Take a look at these graphs of average load times. The Y-axis shows the time to service a particular request. We measure a few different requests and each shows up in a different color. The graph on the left shows performance for the last month and the one on the right shows performance for the year. You should notice a dramatic drop in service times a week and a half ago when HA_Proxy was added for load-balancing.

The graph below shows the same timings, but for actions that authors take. It shows a similar speedup (lower height lines) for authoring. In the graphs for authoring you may also notice a major performance improvement in February of this year. That improvement was thanks to increasing the size of an application object cache.

So why would changing the load balancer have such a big benefit? We were surprised at the magnitude, but not the direction of the improvement. Squid isn't specifically designed to do load balancing, and we were using the Internet Cache Protocol (ICP) to approximate load balancing. Squid would ask frontends to respond to an ICP request and then use the speed at which they responded "no" to determine which one to choose for the request. HA_Proxy is designed to do load balancing. It did take some configuration, but it is working much better than trying to contort ICP for load balancing. Our settings for HA_Proxy choose the front end with the least number of current connections and then choose among equals using round-robin.

As background about our characteristics, Connexions serves about 2 million unique visitors per month. We receive between 50 and 60 requests per second, peaking at around 100 requests per second. Our performance is still spiky. Occasionally a request takes a very long time to serve, and that can be very frustrating for viewers and authors. We will continue to report on performance in the blog as we improve the infrastructure.

New Collection Editor

Thanks to @hedleyroos and @rochecompaan out at Upfront Systems, we're delighted to have a shiny new collection editor. There's a separate post that gives a great overview of how to use it but in this post I'll go under the hood and describe a bit about the technologies used and how they were implemented. Finally, there are a couple of "gotchas" we ran into that are worth discussing.

The editor uses two JavaScript frameworks, ExtJS and jQuery, along with Zope Page Templates (ZPT). Zope is the templating engine used to implement Connexions. The editor uses AJAX (Asynchronous Javascript And XML) to call the server on Connexions. This eliminates the need to refresh the entire page as the old editor did. Users start off with an ExtJS tree that shows how the collection is organized. The tree is reorderable with links to add/delete Subcollections or modules. On load, the collection hierarchy is requested as JSON from a ZPT. JSON (JavaScript Object Notation) is a lightweight data-interchange format.

Every time a module or subcollection is dragged and dropped, a call is made to Zope to update its objects. Since these objects can be moved anywhere in the hierarchy they may get a new parent. To handle this in Zope, we cut, paste, and reorder.

Gotcha 1: Initially the Zope UID was used to keep the Javascript and Zope objects in sync. However, when an object is pasted the UID changes. In the code both the UID and path to the object (using ids) were used but in order to support moving the node to a different parent both would have to be updated so we switched to calculating the path on each update.

The editor also has popup dialog boxes for editing properties of the collection. When a button to open a dialog box is clicked an AJAX call is made to Zope and the resulting HTML is injected directly into the page. The HTML that is sent back also contains a Javascript tag that contains all the logic needed for the dialog. Sending back HTML from the server has the advantage of using the Zope internationalization machinery to support other languages and by keeping the Javascript in the same file it is easier to keep track of which code goes with which dialog box.

Gotcha 2: Those pesky corner cases (in form submission, cancellation, and validation).

Since we request HTML from Zope, very little state is stored on the browser. To support canceling a dialog box, we had to first store the default values. When a user enteres some invalid data and submits, Zope does a nice job of rendering the form with how to fix the problem. However, if the user cancels, we needed to scrub the form clean, removing the validation errors as well.

So, that's a quick run through the new Collection Editor and if you're interested in the code, it can be viewed at our trac page.