Rob firstname.lastname@example.org - Scrum Practice Leader
- Moving incrementally and iteratively to Drupal- making improvements as you move bit by bit
- User comments and recommendations served from Drupal, along with comment history pages, article comments pages
- Syncing data to Drupal every 5 minutes-- all content and comments
- Soon, article pages served from Drupal-- running into a few performance problems
- Next: channel pages served from Drupal, third-party services, registration
- We benefit from Drupal sooner by taking this approach; rather than building the whole site in the background and not benefitting until the end, this way we benefit from improved functionality sooner
- "The Economist is so old that the guy who started it had to be painted rather than photographed"
The old way
- 20-30 mil page views, 3-4 million unique visitors per month - lots of performance and scalability issues
- Want to build the foremost destination online for analyzing and debating global agenda; want to bring visitors into that debate; current system isn't enough to support this vision, that's why they moved to Drupal particularly for comments
- Increase publishing volume with user-generated content (more content w/o more costs)
- The old way: custom CMS built on proprietary stack (MS, ColdFusion, Oracle)
- Blogs were originally MovableType, now are all Drupal
- Broken waterfall processes meant frequent fire-fighting
- Needed to be more responsive to change, deliver business value sooner (projects take a long time to deliver value to organization), more sustainable, happier
- Making these changes incrementally and iteratively; "perfect is the enemy of better"
- Looked at OpenCMS, Alfresco, Joombla, met with other newspapers, considered building a custom system, buying a proprietary system, or going open source
- Drupal as strategic fit: community and content publishing, robust development framework, development language, free software
- Strength of Drupal community
- Selling Drupal internally was a challenge: no suit-wearing Drupal sales force
- Attended DrupalCon Boston 2008, networking within community, engaging w/ Lullabot for workshops and training
- Proof-of-concept to reproduce article page in Drupal; how to use CCK fields to make a rich article content type
- 3 million registered users, articles - data migration is daunting
- Manage the move using Scrum - selling it was easy with charts (developing business value sooner and throughout, management can see progress throughout, shining a spotlight on issues/dysfunction and attacking them along the way - risk decreases a lot faster)
- Take requirements, prioritize based on business value: which are the most important to organization, do those first
- Trained management team in Scrum, development team in Drupal, then started sprinting with help from consultants (2-week sprints, delivering something of value at the end)
- "Maybe not the largest Drupal project, but the most expensive" - lots of consultants
- Proxy approach: Drupal sends JSON over HTTP back and forth with Existing ColdFusion system
- Using native Drupal comments; comments have to be attached for nodes - there has to be a node for every piece of content on the legacy system
- Create nodes on the fly for every ColdFusion request that comes in
- Notion of proxy nodes is a pattern that comes up during integration of Drupal with other systems
- Voting API votes used for recommends; these are also attached to proxy nodes
- Started with proxy approach only; then moved to doing some with subdomain approach - hope to be doing neither soon after moving entirely to Drupal
- Migrating and syncing data every 5 minutes - don't wait until the end to figure out that piece
- Table Wizard and Migrate modules
- Table Wizard writes Views integration for MySQL tables
- Migrate lets you migrate certain views, push into Drupal as nodes/users/taxonomy terms/etc
- Client is involved in how legacy data gets organized in Drupal
- Sat down with client to browse through content and decide what data needs to be moved and what it means
- Migrate keeps track of everything you've done, gives you a dashboard, tells you how far along you are - keeps a mapping table, legacy ID, you can check and see what came across and fix things; does your bookkeeping for you
- Drupal expects to have all the info it needs in its database; something getting published in Oracle needs to be in Drupal promptly - synchronization
- How did you decide what to put into Drupal first?
- Business value: comments, user profiles, recommends
- How many Drupal servers does it take to scale that big?
- Not entirely sure how many servers we have; let's say +/- 12
- Master MySQL server, a few slave MySQL servers - more important aspects have to do with Pressflow
- Pressflow = high performance variant of Drupal 6, completely API compatible with Drupal, but it takes some patches that are in Drupal 7 and moves them in to Drupal 6
- Use Varnish's full capability; Varnish = reverse proxy server, takes load off Drupal/PHP/MySQL
- How do you stop people from trying to shove their emergencies into Scrum process?
- Don't want people going directly to the team like they traditionally do
- Team, Scrum Master, product owner - customer, person who represents the client, has to have power to make decisions on behalf of organization, responsible for managing stakeholders
- Product owner comes to team w/ prioritized list of features for next sprint
- Had two teams in New York and one team in London all doing 3-week iterations in parallel
- Split up site into component parts: profiles, article pages, channel pages, had three product owners who had to manage stakeholders
- Works reasonably well; now we're doing two teams, one system that shows what all teams will do; someone has to keep "product backlog" in order, stopping people from shoving in their "one little thing"
- Base theme is 960 px grid - laying out themes as a series of columns, all sections have to fit into the grid
- Selenium for "user journey" testing; building environments to help manage configurations
- Continuous integration using Hudson - needed a shared place where user tests could run
- Set of servers running on Amazon; Hudson sets off user tests every time there's a commit to the SVN repository
- Apache SOLR search hosted by Acquia- 100,000k articles that have to be available through site search
- People were unhappy with relevance of matches in old site search
- Acquia's hosted search service: really fast, good results
- Apache SOLR: can start filtering results further and further - faceting
- "How do I get SOLR running on my website?" - can self-host, but we went with Acquia
- Other tools for managing people/process?
- In Scrum, less about resource management - we just want dedicated co-located teams, don't worry about availability because of multiple projects: single focus
- Redundancy of function - generalizing specialists, specialists can create bottlenecks/risks
- "How many people need to be hit by a bus before your project fails?"
- Use Google Docs a lot - project backlogs are all spreadsheets, a big wiki, project dashboards that "radiate information to the rest of the organization"
- Focus is on people, not tools
- Test-driven development, writing tests first can sometime be hard with Drupal
Impediments to progress
- Previous processes/structure/culture: command and control - hard habit to break
- Project manager telling people what to do and when to do it by - this is bad management; it has an impact on people
- We want self-organizing teams
- Previously, black box development: low visibility during the project process
- For Scrum, everything needs to be transparent, frequently inspect outcomes, adapt as we go - can't have a postmortem after everything's done, need to do that every day
- Hero developers who go off and solve problems heroically aren't compatible with Scrum
- Previously, developmental silos - departments based on function, these have been removed, but people still want to exist within their old silos
- People want to work on multiple projects like they used to, rather than working on a single project in a dedicated manner
- Previously, traditional line management: where you stack up in the line doesn't matter now, this was a big change
- Engineering practices (specifically quality) - big issue; Scrum is a wrapper for your existing engineering practices, doesn't say anything about testing
- Scrum assumes your engineering practices are great, or you'll make them great quickly
- You can say "we're going to do Scrum" but old habits die hard - focusing on what "done" means and providing a deliverable at the end of each sprint, have to deliver quality too-- have to go live successfully
- Want to deliver "potentially shippable code" at the end of each session - have to have a testing environment that's representative of live environment; been bitten by differences in configuration
- Everything has to be identical in the test environment (just with a scaled down number of servers) - same data center, same network issues, etc
- Hard to bite the bullet on the costs involved in building a testing environment, but it's important
- Hard to simulate kinds of traffic you get in production - plus, have to keep track of session cookies
- Form fields can hurt you - replaying post requests
- Cron jobs that run all the time - cron jobs can stack up and site starts to decay
- Migration of real-time data: code changes are easier to migrate than content changes, what's the process for moving bits of content from development to production?
- When there's content you need to work on for a while before it goes live, work on the live servers but make sure end-users can't see it
- Can use the unpublished flag on a Drupal node to do that; use "views" to see everything unpublished in sports category
- For a small team, that's a reasonable solution
- For bigger organizations with a lot of people working together, use "Workflow" module - nodes step through a series of states
- If it's a business requirement that content has to start off on staging servers and only then push to live, use module "Deploy" - push-button way to push nodes and their dependencies-- users, taxonomy terms, etc-- to another environment
- Technical reason for using external searching - why use SOLR at all? What about Drupal search?
- Drupal 6 is better than previous search mechanisms, but falls apart at a certain scale
- Slow queries, sub-optimal results
- A lot of non-Drupal people have worked on Apache SOLR, Drupal has integrated it well
- Self-hosting, or with Acquia - if you have the talent to run Java apps in your data center and keep it running, self-hosting is a great idea; will reduce latency
- Most of us are struggling to keep PHP/MySQL up as it is, this is where Acquia comes in
- Acquia service is pretty much plug-and-play
- Built-in search doesn't come with facets; can add on facets with the "Faceted Search" module
- SOLR is an enterprise search system; used by Netflix, Expedia, etc.
- Could you use Views instead of facets?
- There's a lot of overlap there, and different possible approaches.
- Full-text searches need SOLR rather than Views
- Some of the wins you've had with Scrum/Drupal, and some weaknesses
- Wins by development teams - prefer this way of working, where business people are only concerned with relative priority of requirements, have no say in how long it takes to implement
- Product owners prioritize "stories", developers size those stories relative to each other, rather than in hours of effort
- Stops the cycle of cutting corners on quality in order to get it done in a shorter timeframe
- Can't get productivity gains w/o changing the way you work
- Product owners need to be involved, can't change requirements mid-sprint
- Have "working agreements" - a kind of social contract
- Scrum isn't a prescription - you can pick and choose the parts that you want that meet your organization's needs
- Specific processes layered on top of simple framework of transparency, working together, and adapting to testing results, can vary
- When will the Economist be fully on Drupal?
- Description says "this month" - that was the plan
- People paying the bills get to make decisions; is it most important for us to go all-Drupal ASAP, or extend functionality of site to be competitive?
- Recent decision was for the latter
- Don't know when