How We Build the OpenStax College Books

In case you are not familiar with OpenStax College (OSC), it is a part of our OpenStax family that publishes free, open, peer-reviewed, commercial quality textbooks. The initial goal is to create textbooks for the top 20 Community College courses in the US. We have 9 books completed, 4 in production and funding for the remaining books.

OSC books are available in print, so there was a technical challenge to generate a print quality PDF for each of the books.  Our earlier PDF generation pipeline uses Latex which results in nice black and white PDFs, but did not meet the OSC requirements.  To meet the OSC requirements, we are using HTML5, CSS3 and a commercial product called PrinceXML. PrinceXML is the only commercial product used by OpenStax CNX. Each of the books has a different design and collation requirement so CSS has to be created for each unique element. The books also share features so we have tried to structure the CSS so it relies heavily on the Cascading part of Cascading Style Sheets.

Another requirement is that we must build the PDFs without human intervention.  It must be totally automated.  Since users can derive copies of the OSC and modify them, we also must deal with missing or added content from the original.

Code Structure

Our CSS code is broken down into 2 main parts: Slots and Skeletons. Skeletons define the namespaces used in a book.  Slots are used to define the styles for the namespaces. Some books are derived from others so their Slot and Skeleton files could be smaller than other books. There are also other CSS files that control numbering (using counters), page formatting and utilities.

PDF Generation Workflow

The workflow is

• Content is stored as XML (CNXML) in OpenStax CNX
• The XML is converted into HTML using a highly modified version of the Docbook transform
• The resulting HTML plus CSS is passed to PrinceXML which generates the PDF

PDFs are generated when books are published or the content inside the book is updated.  All of the front matter is entered by hand into the PDF, but the remainder of the book is auto-generated.

Rewrite Technologies

Our team has been hard at work rewriting the OpenStax CNX site.  A frequently asked question is "What tech are you using?". This post covers a high level overview of the new site.

Architecture

Rewrite is a Single-Page App which means most of the logic lives in a Javascript client. The Client accesses data via REST APIs that are written in Python.  All of our data is stored in a PostgresSQL database. The details of the separate components follow.

Webview

Webview is the Javascript Client.  The basis of the Client is Backbone.js and Bootstrap.  We use several other Javascript packages as well.

• Handlebars.js templating
• Underscore.js
• Require.js

Most of the Javascript is written in Coffeescript and compiled to Javascript.  CSS is developed using Less.

Webview requests json from the Archive via REST APIs.  The json contains the HTML for the content and any metadata. The Client parses the json and displays it.

Editing is a separate view in Webview.  When the user selects to edit, the views are swapped out. The new editor is based on the open-source HTML5 editor Aloha.  We have added several plugins to Aloha that are textbook editing specific.  The development on the editor was done by our team and the OERPub team with OERPub doing the bulk of the work.

We are using Nginx as our web server.

Archive

Archive stores published content and handles search. Content retrieving and search are handled via APIs written in Python. When content is requested, the json is built via stored procedures in Postgres. The json is built using the json functions in Postgres.

Search is done with optimized queries.  We are caching subject and one word searches long term and all searches short term to improve performance.  When a user pages through search results, the cached result is used to populate the next page.

Archive will run on an WSGI compatible server.  We are currently using Waitress as our server.

Publishing

The publishing application integrates with the Archive database. It allows users and third-party applications to publish content to the Archive, where it can be read and distributed to the public. Publishing is built similar to Archive, but differs in many ways. Archive is a read-only content API. Publishing provides an additional set of APIs that handle the publishing workflow, which includes user interactions like license and role (e.g. author or translator) acceptance, as well as the triggering of export files. Users of the OpenStax CNX system will typically never directly interact with this application. Almost all of the backend business logic is handled within the publishing application.

Authoring

Unpublished content is stored in Authoring in Postgres.  Authoring also has APIs used by the Editor and the Workspace. When content is published, an EPUB is generated and passed from Authoring to Publishing.  The EPUB format was selected to pass information between components because it encapsulates all of the info needed.

The Workspace is a listing of all content a user has access to edit. Books and Pages can be created in the Workspace and they can be deleted as well.

OpenStax Accounts

Users are now stored in a shared accounts component.  This was developed so users could have the same account on all OpenStax sites. All CNX users have been migrated to the new Accounts. Accounts uses OAuth so users will also be able to log in using Google, Twitter and Facebook. CNX will no longer create CNX user accounts.  New users will need to use one of their existing OAuth accounts to log in.

Logging

We are currently logging information, errors and user interactions to Syslog.  Our goal is to load the logs into Graphite so we can visually see how our site is being used.

Transformation Services

Transformation Services generates Export files(PDF, EPUB, Zip, etc.) and imports content to the editor. The initial design uses the same import and export code from Legacy CNX inside of a messaging system wrapper.

The messaging using RabbitMQ and several messaging queues. The queues will give us persistence of the file generation requests.  The requests will be sent by Publishing after content has been added to Archive. Publishing will pass an EPUB to Transformation Services that contains all of the data needed to generate the files.  

OpenStax CNX is a deceptively complex site that has many moving parts.  Our goal with this architecture was to design a component based system that can be easily updated and tested without impacting all of the site. All of our code is on Github.

Many thanks to CNX team members Michael Mulich and Derek Kent for reviewing and contributing to this post.

Tech Behind Search Improvements

Over the summer, we have made several improvements to the new version of the site (Rewrite),. One of the biggest improvements was with search. Since we released Rewrite, the performance of search has been a concern.  We did little to improve the speed before the release because of time constraints. It was decided that we needed to revisit it this summer to get things in great shape for the Fall semester.

We implemented the following changes:

• Cache single word and subject searches - we added Memcached to store some searches.  A cron job is reloading the subject searches into the cache on a regular basis.
• Pagination on Search Result page - previously we were displaying all of the search results on the same page. This caused a long page render time which made the slow search even slower. We are now displaying 10 items on each page. Books are displayed first since most users are looking for books. When possible, we use the cached search results to return the next page.
• Improved SQL performance - we tweaked the SQL used for the queries to optimize them.

All of these changes are no-brainers, but have vastly improved search.

OpenStax CNX Development Tools

Over the last couple of years, we have changed our internal development tools. We do our own version of Agile development and have found these tools best meet our needs.

For Sprint planning, we use Trello.  Our team members are in many locations, so having a web-based tool to outline our Sprints has been very important. We create cards for User Stories or issues and work from the boards for Sprint planning and working on Sprints.

Our code is stored in Github. We previously ran our own SVN server, but slowly migrated all of our code to Github. It is a great tool. Our workflow for using Github is

• Each component has a separate Repository.
• Each Repository has a Master branch.
• Each Repository has a production branch that contains the code currently in production.  This allows us to continue working and merging to Master, but also be able to fix problems in production easily from the production branch,
• Developers branch off Master and code the Trello card they are working on. Once the code is completed and unit tested, the developer creates a Pull Request in Github. The Pull Request is to merge the code into Master.
• A Pull Request triggers a code review by another team member.  Code reviews generally result in a review of the code as well as a manual test of the code.
• Pull Requests are also unit tested using automated testing via Travis-CI
• Once a Pull Request is approved, it is merged and the branch is deleted from Github.

Most of our meetings are held on Skype.  Skype has the simplicity of making a phone call and is mostly reliable.  We also use Google Hangouts when we need to share code or other screen sharing. It works really well, but if not as easy to start up as a Skype call.

Our team relies on IM. We have a Jabber server that some of the team uses and others use Google Talk or Hangouts.  IM is our virtual hallway and is a key part of our communication.

Publishing Added to OpenStax CNX Demo Site

We have recently added publishing to our Demo site.  If you are not aware, Demo is our alpha testing ground for the new version of our site,  Not everything is working yet and there are bugs remaining, but users can get a feel for what is coming soon to OpenStax CNX.

Users can

• Create, edit and publish a new Page
• Create, edit and publish a new Book 
• Derive a Page or Book when viewing content
• Derive a Page when editing a Book 
• Publish a Book with published and unpublished Pages (the unpublished Pages get published with the Book). 

Please take a look and give us some feedback at techsupport at cnx dot org.  We would love to have you help us improve editing and publishing. The site will be updated regularly so check back often to see our progress. 

OpenStax College Android App Released

A new Android app that features the OpenStax College books was released on May 2nd.  The features include

• Viewing released books
• Saving Books or Pages to your Bookmarks
• Take notes while reading
• Export your notes to a text file or share them.
• Share a Book or Page

The app is open source and the code is in Github. You can install the app from the Google Play store.

The app can be used to access these books

• College Physics
• Introduction to Sociology
• Biology
• Concepts of Biology
• Anatomy and Physiology
• Introductory Statistics
• Principles of Economics
• Principles of Microeconomics
• Principles of Macroeconomics

From a technical point of view, the app is based off of the existing OpenStax CNX Android app.  As part of our name change, the mobile version of our site was modified to work better with the Android app.  The result is a simplified look of the content (see screenshot).

If you use the app, we would love to hear feedback either as a rating of the app or by contacting us at android at cnx.org.

OpenStax CNX Featured On Floss Weekly Podcast

Two OpenStax team members were featured guests on the Floss Weekly podcast this morning.  Floss Weekly covers open source software for the TWIT podcast network.  Kathi Fletcher and Ross Reedstrom told the history of the project along with a discussion of the work we are currently doing on OpenStax CNX. The podcast is available to view or download from TWIT.

New Name and New Editor Demo

In case you missed the announcement at the Connexions Conference this week, Connexions has been renamed to OpenStax CNX!  We have also renamed this blog to OpenStax CNX Developer Blog. You can visit us at openstaxcnx.org. The renaming give us a unique presence on the web and helps us build a family of OpenStax sites to provide open options for education.  Our other sites are available from openstax.org.

At the 2014 conference, we released a beta version of our new editor. It can be found at demo.cnx.org. Log in using your CNX, Facebook, Twitter or Google accounts and give the new editor a try.  We would love to hear your feedback as we continue to work on it.  It is definitely beta software so expect to find bugs.  We will continue to update it we get ready for the public release at the end of June.

This blog has been dormant for the past few months which is not good.  Our plan is to start posting regularly again since we have lots of cool code happening in the OpenStax College textbooks and in the rewrite of OpenStax CNX.

Connexions Conference Sprints

We are holding 2 days of Sprints on April 24 and 25 following the 2013 Connexions Conference which is on April 23.  The sprints are an opportunity for you to work directly with the Connexions team, contribute your time and effort to Connexions and to have fun.  You don't need to attend both days.  Show up and leave as you like.  We have tasks for all skill levels and you don't need to be a programmer. 

The list of tasks is on Trello. 

 Just click on a card (white box) to see the details of each task.  If you are attending the conference, we hope you will stay for the Sprints.  If you are in Houston, but not attending the conference, you are welcome to join us at the Sprints.

Jenkins forces reinstall of server images on Rackspace

This is much more DevOps than Dev, but it is important as well ! One difficulty we have doing remote development with a team spread over at least a million cities (I haven't actually counted) is that the local development LAN for Bob is frankly inaccessible for Alice and Charlie. Maintaining n-1 VPN links is just going to drive us all mad.

So the simple solution is to rent a couple of cloud servers and host the development builds on there. The cost is pretty minimal, and it seemed good - until I realised that Jenkins, our friendly Continuous Integration Butler, was unable to re-install a cloud server from an image. Or rather it *could* - but Jenkins did not know when the server was rebuilt, and the server itself, which could signal jenkins, would have no state, so could not signal jenkins with *why* it had been rebuilt. So, Pantry Bell (https://github.com/lifeisstillgood/pantrybell). We simply run a webserver, that when called (perhaps from one Jenkins job) will force a re-imaging of a Rackspace server. This has a simple callback in rc.local (wget http://jenkins.frozone.mikadosoftware.com/pantrybell//`hostname`) So - a server forces a rebuild, on successful rebuild, the server calls home, and then the "real" jenkins job can be triggered using the Jenkins API calls. Amazingly enough it works, has trivial security, no REST compliance but it works

OpenStack and Rackspace

OpenStack and Rackspace

I suspect that if you find this as exciting as I do, one or both of us need to go out looking for a life. However, OpenStack is nearly here, and Connexions is in the Early Access Program. WooHoo.

OpenStack is in the frankly hilariously named Essex release, and is a collaboration between NASA (yes!) and Rackspace (and now 150 other companies) all of whom bascially want some of Jeff Bezos’ AWS lunch.

Now, I am currently trying to get Jenkins to nicely play with my cloud servers (see Pantry Bell) and it will use the pythonic nova Real Soon Now.

But the real pull is that we will be using rackspace to integrate all the different components involved in the new system, early as possible. The system currently builds the repository and the (old) editor, and I am working on bringing in the new Aloha editor and the webview asap. The sooner we see everything working together the sooner we shall find those great moments of code serendipity.

We have an opportunity as a bunch of Open Source developers to use the first real F/OSS cloud based offering and even influence a little part of its development. I am excited to see where it will go, and excited to see if it can provide measurable benefits to Connexions, either in development simplicity or sysadmin time savings. Interesting Times are ahead.

The Usual Suspects

Starting Blogging !

This is the first of hopefully a regular series of blogs from the developers behind Connexions.

Connexions

Connexions provides free (beer and speech) educational resources to students and teachers around the world. It has developed over ten years a eco-system of authoring tools, repository and means to serve the textbooks, exercises, snippets of knowledge to 2 million people a month.

And they are building a major update of that code.

And its all Open Source - and being developed right in front of your eyes. (Mail us if you want to join in.)

And thats what we are doing here ...

The people¶

(If your gravatar is not here, please let me know the right email address to use)

Jessica Burnett		Paul Brian		Ed Woodward
Philip Schatz		Michael Mulich		Marvin Reimer
Ross Reedstrom

Google Summer of Code Students

Debajyoti Datta		Saket Choudray		Alasdair Corbett
Yanchai Ye

The code:

https://github.com/Connexions

The plan

I am hoping we can link to the specs, and have the developers involved in each component post about their updates or changes on a regular basis.

We shall see, as the old man said.

Connexions Development Update 2-3-2012

Well, the dream of posting a weekly update is not working out.  Things are very busy here and it is a challenge to find the time to write this post.  I am committed to writing an update periodically so I'm going to modify my plan to make this bi-weekly.

CCAP

• The Physics rendering in Prince XML is complete except for some math issues.  The code is on the development server and should move to production soon with Sociology.
• Our current focus is on getting the Sociology rendering tested and released.  The current release date is Feb. 8th or 9th.
• There are several Math issues in the Physics book.  Some of the problems were caused by the Word importer, some by the original structure of the math in the Word documents and others are font issues.  When the Physics book is migrated to production, we will run a script to clean up some of the import problems to minimize the need for human intervention.  Some of the font issues will not be able to be resolved because the Stix fonts do not have the needed font.  An example is an italicized delta.

CNX Conference Preparation

• We have over 50 people signed up for the Sprints!  We are very excited about this.  As best we can tell, about 30 of these will be developers/coders/designers.  With this many people involved, we are going to make a huge effort to have clear easy install instructions for the various options.  Next week, we will be testing everything and finalizing the instructions.
• Once the instructions are complete, we will post the link on the Rhaptos list so you can install Rhaptos or anything else you might need prior to arriving at the conference.
• Many thanks to all that are planning to Sprint with us!

OERPub API

• We released the latest version of the OERPub API last week.  It will be used during the Conference Sprints by the sprinters working on content.

HTML Editor Discussion

• We have had a little time to continue our discussion in house regarding a new editor for Connexions.
• Most of the discussion has been Microdata vs Microformats.  Neither has much traction in the wild.  
• CSS3 seems to have support for Microdata which would allow us to use it to decorate semantic elements.  Microformats also have CSS support.
• There has also been discussion on which editor to select.  TinyMCE and Aloha have been our focus.
• TinyMCE has a nice UI, but does not support all of HTML5.
• Aloha supports all of HTML5, but has an odd UI.  The UI is not bad, just takes a little getting use to.  The demos on Aloha's site are very specific so we need to install it and play with the configurations of the UI.
• I'm sure the editor will be discussed at the conference so let us know your thoughts if you are attending. 

Connexions Dev Update for Jan. 20, 2012

Lots of info to share since last week's update was missed.


CCAP

• We tested the Physics book PDF generation and the Word Importer changes in QA last week.    The code was using FOP and Docbook to create more professional looking PDFs.  Our plan was to release this code and in a couple of weeks release the Prince XML version of the PDF generation.  The urgency was caused by the lack of harddrive space on the development server where the content is being created.  We have now fixed the space issue so it was decided not to release the FOP version of the PDF code.  We are focused on the new Prince XML version.
• Documentation - We have some new styles needed for the Sociology book to our documentation of the new markup.
• Coding is continuing on the PrinceXML version of the PDF.  You can see a sample of the Sociology layout at http://mountainbunker.org/~schatz/sociology.pdf.  It has Physics content, but the color scheme of Sociology.
• Some minor issues have been brought up by our content team regarding Math display in the Physics book.  Some of this will be fixed in the Prince XML code, but others will have to be fixed when we do the migration to production by tweaking the MathML generated by the Word Importer.

CNX Conference Preparation

• As the CNX Conference approaches, we are preparing for the sprints that will happen after the conference.
• The Bug List for the Sprint has been completed.
• Documentation for the Sprint has started.

Donate Button Change

• The donate button now has a minimum donation of $1.  The original minimum was $10. We hope this will lead to more donations.

iPhone App

• An Apple Developer Program for iOS was finally purchased last week.  It has taken months for all the documentation problems to get ironed out.

Consortium Tech Committee Meeting

• The Connexions Consortium Tech Committee held its monthly meeting last week.
• Discussions included current Connexions dev work, OERPub API update, Sprint topics and other member updates.  Check the minutes for more details.

OERPub API

• We are currently testing a new release of the OERPub API which uses Sword to add and modify Connexions content.
• The latest release allows users to add content to a Lens, add Featured Links to a module and to create a collection in a Workspace or Workgroup.  The code should be released early next week.
• The latest documentation for the API is at https://trac.rhaptos.org/trac/rhaptos/wiki/TechnicalDocumentation/Code/Sword

HTML Editor Discussion

• We have been talking within our team and on the Rhaptos list about the possibility of using HTML5 Microdata in the new WYSIWYG editor to document non-HTML elements in CNXML in the editor.
• HTML5 has some traction because of support from schema.org, but it also has problems such as very limited browser support.
• The thread on the Rhaptos list has more information.  

Weekly Dev Report for Jan. 6, 2012

As part of an effort to increase our transparency, the dev team at Connexions will be posting an update on the projects we are working on.  This is our first report and we will make a best effort to do an update every Friday.  We will not be detailing everything we are working on, just the highlights that should be of interest to the Connexions Community.  Feel free to comment here on the blog or on the Rhaptos Development mailing list.

This week the team returned to work after a Holiday break.  Our work included:

• CAP Code Testing - CAP (College Access Project) is a rebuilding of our PDF generation system to use Docbook instead of LaTeX.  We were testing the single column textbook style on a development server prior to our break.  This week, we moved the code to QA (merged to trunk and created eggs).  Testing should finish up early next week.  The code will be moved to production, but will not be used by any collections yet.  You can view a several chapter sample from the code here.
• CAP Double Column PDF - Prior to our Holiday break, we determined that Docbook cannot handle the 2 column layout that is required for some of the CAP books.  We reviewed a couple of commercial products (Antenna House and Prince XML) and decided to move to using Prince XML.  Prince XML converts HTML to PDF, so it is a new direction for our PDF generation.  Our plan is to create the 2 column PDF needed in Prince XML and then replace the 1 column Docbook PDF code with code that will work in Prince XML.  Coding of the 2 column PDF has begun.
• Design Discussions - We are planning on moving to a component based architecture in 2012 and are having daily discussions about how to procede.  Part of the complication of this plan has been the lack of an agreement with Google so we know how to move forward.  That situation has stabilized somewhat so our planning is moving forward in ernest.  As soon as we have some details to share, links will be posted on the Rhaptos list and in the weekly update.  We want and need feedback from the Connexions tech community on this project.
• Connexions Conference - We are planning on at least a 2 day sprint as part of the Conference.  A successful sprint requires planning and preparation so that developers can get what they need installed quickly and start coding.  We are testing our Rhaptos install along with Roché's Plone 4 install to make sure they work as needed.  We will also create some VirtualBox appliances so Mac and Windows users can get up and running quickly.  Those of us doing talks are doing slide preparation.  The conference starts Feb. 15.  You can see more details at the conference site.

Please let us know what you think of our first update.  We would like these to be informative, but also a quick and easy read.

PDF Generation Updated

Last week, we released an update to our PDF generation code.  The update was coded as part of the College Open Textbook project.  The update moved some special PDF features that were only in the Collaborative Statistics PDF to all PDFs. 

The two main visible PDF changes are:

• The numbering of Exercises and Examples inside of a section now take the section number as part of their numbering.  For example, if a group of 3 Exercises are in section 2.4, the Exercises will be numbered 2.4.1, 2.4.2 and 2.4.3.  Previously, the Excercises would have been numbered 2.1, 2.2 and 2.3.
•  Sections marked as Homework now have a page break before them.  This allows the user to print out the Homework without any of the text of the book being printed.  There is a side effect of adding more pages to some collections.

Existing PDFs were not updated to use this new formatting.  They will be updated as they are republished by the authors.  If you enjoy browsing LaTeX, feel free to check out the new code.

Connexions Featured in Google Open Source Blog

The Google Open Source blog is featuring organizations participating in Google Summer of Code 2011. Connexions was featured in the July 15th blog post.

Connexions Sprint at Plone Symposium East

We are holding a coding sprint as part of the Plone Symposium East. Developers from across the globe have come together to improve Plone, the software Connexions is built on, and to improve Connexions itself. We will be working today and tomorrow if you want to join us. Info about the sprint is at rhaptos.org. Click on one of the links at the top of the page for details.











Coders gather to discuss sprint options.

Connexions Sprint participants at work. We have developers from US, South Africa, Sweden, Mexico, the UK and Vietnam working on Connexions today. Most are in the room, but some are working with us remotely.

New Release of Connexions for Android

Version 1.5 of Connexions for Android has been released. It contains cosmetic and usability changes along with a bug fix. The changes include:

• Search is now a popup window instead of a separate tab. This makes search quickly and easily available on every screen. I also removed the Google search. Only Connexions search is used now.
• The app is a little more colorful and better looking.
• Added option menus to all of the views. List views previously only had context (long press) menus. The long press menus are still there and necessary for some operations.
• The Lens tab is now the Content tab. Added Connexions Featured Content to the Content tab to allow new users to quickly find something to view. I will probably add the new content feed to this tab at some point.
  
• Added a one time popup Toast message that explains a little about using the app. It is only displayed the first time a user opens the app after installing it.
• Fixed a bug that caused the app to force close when the orientation was changed when viewing a list of lens items.

I made most of these changes after reading the Apple User Interface Guidelines in anticipation of writing an iPhone app for Connexions. The previous versions of the Android app were made with functionality in mind, not looks. If you have tried the app in the past and were unimpressed, please try it again and give us some feedback. I want to continue to improve it.

As always, the code is open source and is in our repository. You can download the app directly from Connexions or from the Android Market. If you try the app and find it useful, please leave a comment in the Android Market.

Version 1.4 of Connexions for Android Released

Version 1.4 of Connexions for Android was released today. It adds 2 new features and fixes a file downloading bug. The new features are a File Management screen that can be used to open or delete downloaded PDF or EPUB files and Pinch and Zoom was added to the web view for devices with Android 2.1 or above. The bug fix corrected a problem with phones rebooting while downloading a large PDF or EPUB file.

As always, the code is open source and is in our repository. You can download the app directly from Connexions or from the Android Market. If you try the app and find it useful, please leave a comment in the Android Market.