Rob robpurdie@economist.com - Scrum Practice Leader
twitter.com/robpurdie
facebook.com/robpurdie

Overview

  • Moving incrementally and iteratively to Drupal- making improvements as you move bit by bit
  • User comments and recommendations served from Drupal, along with comment history pages, article comments pages
  • Syncing data to Drupal every 5 minutes-- all content and comments
  • Soon, article pages served from Drupal-- running into a few performance problems
  • Next: channel pages served from Drupal, third-party services, registration
  • We benefit from Drupal sooner by taking this approach; rather than building the whole site in the background and not benefitting until the end, this way we benefit from improved functionality sooner
  • "The Economist is so old that the guy who started it had to be painted rather than photographed"

The old way

  • 20-30 mil page views, 3-4 million unique visitors per month - lots of performance and scalability issues
  • Want to build the foremost destination online for analyzing and debating global agenda; want to bring visitors into that debate; current system isn't enough to support this vision, that's why they moved to Drupal particularly for comments
  • Increase publishing volume with user-generated content (more content w/o more costs)
  • The old way: custom CMS built on proprietary stack (MS, ColdFusion, Oracle)
  • Blogs were originally MovableType, now are all Drupal
  • Broken waterfall processes meant frequent fire-fighting
  • Needed to be more responsive to change, deliver business value sooner (projects take a long time to deliver value to organization), more sustainable, happier
  • Making these changes incrementally and iteratively; "perfect is the enemy of better"

Why Drupal?

  • Looked at OpenCMS, Alfresco, Joombla, met with other newspapers, considered building a custom system, buying a proprietary system, or going open source
  • Drupal as strategic fit: community and content publishing, robust development framework, development language, free software
  • Strength of Drupal community
  • Selling Drupal internally was a challenge: no suit-wearing Drupal sales force
  • Attended DrupalCon Boston 2008, networking within community, engaging w/ Lullabot for workshops and training
  • Proof-of-concept to reproduce article page in Drupal; how to use CCK fields to make a rich article content type

Using Scrum

  • 3 million registered users, articles - data migration is daunting
  • Manage the move using Scrum - selling it was easy with charts (developing business value sooner and throughout, management can see progress throughout, shining a spotlight on issues/dysfunction and attacking them along the way - risk decreases a lot faster)
  • Take requirements, prioritize based on business value: which are the most important to organization, do those first
  • Trained management team in Scrum, development team in Drupal, then started sprinting with help from consultants (2-week sprints, delivering something of value at the end)
  • "Maybe not the largest Drupal project, but the most expensive" - lots of consultants

Integrating CMS's

  • Proxy approach: Drupal sends JSON over HTTP back and forth with Existing ColdFusion system
  • Using native Drupal comments; comments have to be attached for nodes - there has to be a node for every piece of content on the legacy system
  • Create nodes on the fly for every ColdFusion request that comes in
  • Notion of proxy nodes is a pattern that comes up during integration of Drupal with other systems
  • Voting API votes used for recommends; these are also attached to proxy nodes
  • Started with proxy approach only; then moved to doing some with subdomain approach - hope to be doing neither soon after moving entirely to Drupal

Migrating data

  • Migrating and syncing data every 5 minutes - don't wait until the end to figure out that piece
  • Table Wizard and Migrate modules
  • Table Wizard writes Views integration for MySQL tables
  • Migrate lets you migrate certain views, push into Drupal as nodes/users/taxonomy terms/etc
  • Client is involved in how legacy data gets organized in Drupal
  • Sat down with client to browse through content and decide what data needs to be moved and what it means
  • Migrate keeps track of everything you've done, gives you a dashboard, tells you how far along you are - keeps a mapping table, legacy ID, you can check and see what came across and fix things; does your bookkeeping for you
  • Drupal expects to have all the info it needs in its database; something getting published in Oracle needs to be in Drupal promptly - synchronization

Questions

  • How did you decide what to put into Drupal first?
    • Business value: comments, user profiles, recommends
  • How many Drupal servers does it take to scale that big?
    • Not entirely sure how many servers we have; let's say +/- 12
    • Master MySQL server, a few slave MySQL servers - more important aspects have to do with Pressflow
    • Pressflow = high performance variant of Drupal 6, completely API compatible with Drupal, but it takes some patches that are in Drupal 7 and moves them in to Drupal 6
    • Use Varnish's full capability; Varnish = reverse proxy server, takes load off Drupal/PHP/MySQL
  • How do you stop people from trying to shove their emergencies into Scrum process?
    • Don't want people going directly to the team like they traditionally do
    • Team, Scrum Master, product owner - customer, person who represents the client, has to have power to make decisions on behalf of organization, responsible for managing stakeholders
    • Product owner comes to team w/ prioritized list of features for next sprint
    • Had two teams in New York and one team in London all doing 3-week iterations in parallel
    • Split up site into component parts: profiles, article pages, channel pages, had three product owners who had to manage stakeholders
    • Works reasonably well; now we're doing two teams, one system that shows what all teams will do; someone has to keep "product backlog" in order, stopping people from shoving in their "one little thing"

Features

  • Base theme is 960 px grid - laying out themes as a series of columns, all sections have to fit into the grid
  • Selenium for "user journey" testing; building environments to help manage configurations
  • Continuous integration using Hudson - needed a shared place where user tests could run
  • Set of servers running on Amazon; Hudson sets off user tests every time there's a commit to the SVN repository
  • Apache SOLR search hosted by Acquia- 100,000k articles that have to be available through site search
  • People were unhappy with relevance of matches in old site search
  • Acquia's hosted search service: really fast, good results
  • Apache SOLR: can start filtering results further and further - faceting
  • "How do I get SOLR running on my website?" - can self-host, but we went with Acquia

Questions

  • Other tools for managing people/process?
    • In Scrum, less about resource management - we just want dedicated co-located teams, don't worry about availability because of multiple projects: single focus
    • Redundancy of function - generalizing specialists, specialists can create bottlenecks/risks
    • "How many people need to be hit by a bus before your project fails?"
    • agilemanifesto.org
    • Use Google Docs a lot - project backlogs are all spreadsheets, a big wiki, project dashboards that "radiate information to the rest of the organization"
    • Focus is on people, not tools
    • Test-driven development, writing tests first can sometime be hard with Drupal

Impediments to progress

  • Previous processes/structure/culture: command and control - hard habit to break
  • Project manager telling people what to do and when to do it by - this is bad management; it has an impact on people
  • We want self-organizing teams
  • Previously, black box development: low visibility during the project process
  • For Scrum, everything needs to be transparent, frequently inspect outcomes, adapt as we go - can't have a postmortem after everything's done, need to do that every day
  • Hero developers who go off and solve problems heroically aren't compatible with Scrum
  • Previously, developmental silos - departments based on function, these have been removed, but people still want to exist within their old silos
  • People want to work on multiple projects like they used to, rather than working on a single project in a dedicated manner
  • Previously, traditional line management: where you stack up in the line doesn't matter now, this was a big change
  • Engineering practices (specifically quality) - big issue; Scrum is a wrapper for your existing engineering practices, doesn't say anything about testing
  • Scrum assumes your engineering practices are great, or you'll make them great quickly
  • You can say "we're going to do Scrum" but old habits die hard - focusing on what "done" means and providing a deliverable at the end of each sprint, have to deliver quality too-- have to go live successfully
  • Want to deliver "potentially shippable code" at the end of each session - have to have a testing environment that's representative of live environment; been bitten by differences in configuration
  • Everything has to be identical in the test environment (just with a scaled down number of servers) - same data center, same network issues, etc
  • Hard to bite the bullet on the costs involved in building a testing environment, but it's important
  • Hard to simulate kinds of traffic you get in production - plus, have to keep track of session cookies
  • Form fields can hurt you - replaying post requests
  • Cron jobs that run all the time - cron jobs can stack up and site starts to decay

Questions

  • Migration of real-time data: code changes are easier to migrate than content changes, what's the process for moving bits of content from development to production?
    • When there's content you need to work on for a while before it goes live, work on the live servers but make sure end-users can't see it
    • Can use the unpublished flag on a Drupal node to do that; use "views" to see everything unpublished in sports category
    • For a small team, that's a reasonable solution
    • For bigger organizations with a lot of people working together, use "Workflow" module - nodes step through a series of states
    • If it's a business requirement that content has to start off on staging servers and only then push to live, use module "Deploy" - push-button way to push nodes and their dependencies-- users, taxonomy terms, etc-- to another environment
  • Technical reason for using external searching - why use SOLR at all? What about Drupal search?
    • Drupal 6 is better than previous search mechanisms, but falls apart at a certain scale
    • Slow queries, sub-optimal results
    • A lot of non-Drupal people have worked on Apache SOLR, Drupal has integrated it well
    • Self-hosting, or with Acquia - if you have the talent to run Java apps in your data center and keep it running, self-hosting is a great idea; will reduce latency
    • Most of us are struggling to keep PHP/MySQL up as it is, this is where Acquia comes in
    • Acquia service is pretty much plug-and-play
    • Built-in search doesn't come with facets; can add on facets with the "Faceted Search" module
    • SOLR is an enterprise search system; used by Netflix, Expedia, etc.
  • Could you use Views instead of facets?
    • There's a lot of overlap there, and different possible approaches.
    • Full-text searches need SOLR rather than Views
  • Some of the wins you've had with Scrum/Drupal, and some weaknesses
    • Wins by development teams - prefer this way of working, where business people are only concerned with relative priority of requirements, have no say in how long it takes to implement
    • Product owners prioritize "stories", developers size those stories relative to each other, rather than in hours of effort
    • Stops the cycle of cutting corners on quality in order to get it done in a shorter timeframe
    • Can't get productivity gains w/o changing the way you work
    • Product owners need to be involved, can't change requirements mid-sprint
    • Have "working agreements" - a kind of social contract
    • Scrum isn't a prescription - you can pick and choose the parts that you want that meet your organization's needs
    • Specific processes layered on top of simple framework of transparency, working together, and adapting to testing results, can vary
  • When will the Economist be fully on Drupal?
    • Description says "this month" - that was the plan
    • People paying the bills get to make decisions; is it most important for us to go all-Drupal ASAP, or extend functionality of site to be competitive?
    • Recent decision was for the latter
    • Don't know when