An update on our Drupal conversion

We've pushed into mid-November the relaunch of Jacksonville.com on our new Drupal-based news site management system. We're not concerned about the technology, but we are concerned about the people. Radically changing production processes in the middle of the presidential election didn't seem to be a really bright idea.

I spent most of last week in Jacksonville teaching classes on how to use the new system, and I'm back again this week to help in any way I can. Joe Allen-Black, Jonathan Bennett and others on the team have been doing a great job teaching basic sessions while also continuing to work on site development and their day jobs. Just about everybody in the Times-Union's newsroom has been through at least a 90-minute training session, and many have had advanced training beyond that.

Are we overtraining? It's worth keeping in mind that most of the folks in the newsroom have never had any opportunity to work on the Web, being locked out through organizational and technological barriers.

One surprising thing is how much time we've spent discussing some parts you might think were pretty simple and straightforward.

Take the basic story ("editorial node") editing form, for example. It's just a news story. How complicated could it be?

Well, it turns out that the development team has blended in some very cool enhancements, such as an Ajax-driven (Javascript) tool for finding and creating bidirectional links with related content and a tool for creating stock data links if the story is about a publicly held business. Embedded video handling is very important -- Jacksonville.com actually posts more video than the local television stations. Photo handling and the interaction of the Web system with the newsroom's print-focused legacy DTI content management system is quite sophisticated.

We're making very heavy use of some common Drupal tools that have a steep learning curve but a big payoff: the highly flexible Taxonomy system, the Views family of modules, and the Panels family. Part of the training challenge is equipping editors and senior managers with a deep enough understanding of those tools that they can actively manage the site's future growth and direction.

Back at the server farm in Augusta, a team at the network operations center is taking all this very seriously as well. We're moving from a model in which the server generally worked with flat HTML files to one in which the site is heavily dynamic, with pageviews created on the fly by PHP scripts, potentially highly personalized, and much more expensive in terms of computational horsepower. So there's a whole new cluster of Linux-based servers with an interesting configuration.

Out in front is a server running Squid (a caching server) and Squirm (a redirection/proxy server). We've used Squid for years as an accelerator in our classified systems. This new role is different: Squid/Squirm are going to let us blend new Drupal site content with old flat-file content coming from an array of separate servers, hiding all the details from the outside world, and (importantly) preserving old URLs.

Behind Squid/Squirm will be the old Jacksonville.com server and an array of new boxes running Drupal. One of the strengths of Drupal is that it's designed for linear scalability -- you can just throw more hardware at it when your traffic grows. Drupal also makes use of a lot of fancy database query caching internally, and we're adding Memcached to take that load away from the database and put the caches directly in system RAM.

Then behind Drupal is yet another array of servers: A MySQL database cluster. MySQL is especially good at "read" (as opposed to "write") performance, which is why Google uses it to power it ad-delivery engine. We have Drupal sending all writes to a master server, while reads can come from satellites.

All of this is designed to serve all the Morris newspapers as we go forward, so scalability is critically important.

Notably, every piece of software I've mentioned -- Linux, PHP, Javascript, Drupal, Squid, Squirm, Memcached, MySQL -- is open source.

One (legitimate, I think) criticism of "old media" companies is that they've been unable to get their technology organizations out of the 20th century. While both "two guys in a garage" Web startups and new-media giants like Google are moving quickly with open-source tools, corporate IT departments tend to look to familiar old vendors and even closed systems for solutions. We may make some new mistakes, but we're not going to make that one.