Making Agile User Centered

To integrate UX activities into an agile software development process can be tough. Just consider this discussion between Kent Beck the creator of Extreme Programming and Alan Cooper a leading proponent of Interaction Design.

I’ll divide what Alan is talking about into two things: a set of techniques, and the larger process into which they fit. While I’m 100 percent with the techniques themselves, I’m 100 percent against the process that he described for using them. The techniques are optimized for being thoughtful in a cognitively difficult, complicated area where you’re breaking new ground, and the thinking that’s embedded in the practices is absolutely essential to doing effective software development.

The process, however, seems to be avoiding a problem that we’ve worked very hard to eliminate. The engineering practices of extreme programming are precisely there to eliminate that imbalance, to create an engineering team that can spin as fast as the interaction team. Cooperesque interaction design, I think, would be an outstanding tool to use inside the loop of development where the engineering was done according to the extreme programming practices. I can imagine the result of that would be far more powerful than setting up phases and hierarchy.

To me, the shining city on the hill is to create a process that uses XP engineering and the story writing out of interaction design. This could create something that’s really far more effective than either of those two things in isolation. (Kent Beck)

The techniques that Cooper proposes seek to define the behavior of software products “from the point of view of understanding and visualizing the behavior of complex systems, not the construction of complex systems.” He views “interaction design as part of the requirements-planning or product-planning process rather than as something that comes after the product planning is done.”

The User

Despite the fact that its tough to integrate agile and user centered design approaches it is possible because there are similarities – most notably the desire to create useful software. Take, for example, Alistair Cockburn’s comparison of agile development to a cooperative game:

Software development is a (resource-limited) cooperative game of invention and communication. The primary goal of the game is to deliver useful, working software. The secondary goal, the residue of the game, is to set up for the next game.

And as the name suggests, user centered design is about placing people, users of a piece of software, at the centre of the design process – making software that is useful to them – to do this effectively UX professionals seek to understand user goals and motivation through user research.

Or put another way user centered design helps us understand how people use our products, while agile development let us build and deliver software that is more relevant to its users. Both approaches are trying to deliver meaningful software products to people.

The problem then is not because the two approaches are trying to achieve something different but rather the way of achieving it is different; and specifically the sequencing of events and the time given to different aspect of the product development life cycle. Agile’s iterative development cycle is its key strength, but it also makes for some tight deadlines and people often question whether there is sufficient time to fully consider the users’ needs? However, as Alan Cooper suggests, the time to define the user’s needs is not during the development but before the development starts.

I’ve written previously about using an envisaging team to scout out ahead. Not to design the solution but to understand the problem space – understand what the business needs, what users need, who the users are and what the technology ecosystem looks like – and report this back in a coherent fashion to the rest of the team. This isn’t Big Design Up Front by another name. The envisaging team’s work should be limited to discovery work only. This combined with regular rounds of user testing between each iteration – the results of which are fed into the next round of development – can mean that agile software development can be highly user focused.

Photo: The User, by Pete Ashton. Used under licence.

Living with Code

I’ve been responsible for projects of widely different sizes and complexity. From small production jobs – with no real software development (just visual design and integration) – through to large scale application developments, which we’ve then need to live with, to maintain and develop. And big projects, especially products (that need to be reused), aren’t scaled up small projects. They need to be managed differently not just because the projects are more complex but also because you will need to be able to live with the code for years to come.

Developing a large application isn’t the same as developing a small application. For starters large projects are significantly more complex than small projects – they have more interdependencies, there are more unknowns, indeed more unk unks. A project that is three times as big isn’t three times as complex – its 12 times more complex, at least. As Steve McConnell puts it, just because you can build a doghouse doesn’t mean you can build a skyscraper.

People who have written a few small programs in college sometimes think that writing large, professional programs is the same kind of work-only on a larger scale. It is not the same kind of work. I can build a beautiful doghouse in my backyard in a few hours. It might even take first prize at the county fair’s doghouse competition. But that does not imply that I have the expertise to build a skyscraper. The skyscraper project requires an entirely more sophisticated kind of expertise. The difference in complexity between student programs and professional programs can be just as great, and non-professional programmers -underestimate the difference in required expertise at their own peril.”

And to make matters worse, if you’ve not attempted to build a doghouse, skyscraper or fort before it’s likely that your estimates will be rubbish – since its likely that when planning your work – you’ll miss off important tasks, underestimate unfamiliar tasks, that so of thing. Clearly having team members that have worked on similar applications before helps, as does adopting an agile project management approach.

Agile software development isn’t necessary with small projects – because its reasonable to assume the requirements are stable, there shouldn’t be any unk unks and you’ve probably built a dog house before. But as applications get bigger, as projects become more complex, those assumptions don’t hold true. And you need to able able to manage the new information as it arises. But that’s not the whole story.

The code also needs to be easily maintained by your development team – they need code they can live with – and that doesn’t mean the team that build the application, because if you are building a product that will be used, maintained and developed over a number of years the software engineers that end up looking after the code won’t be the people that wrote it. With small projects – the code can usually be treated as throw away code – there’s no desperate need to supported it.

Clearly some technologies make writing supportable code easier than others. For example, while there is nothing inherently wrong with PHP – indeed it has many advantages, its easy to learn and quick to develop with – its all too easy to end up with spaghetti code, because PHP promotes conflating the business logic and presentation – this can make it painful to maintain, extend or debug. As Tim Bray points out:

…based on my limited experience with PHP (deploying a couple of free apps to do this and that, and debugging a site for a non-technical friend here and there): all the PHP code I have seen in that experience has been messy, unmaintainable crap. Spaghetti SQL wrapped in spaghetti PHP wrapped in spaghetti HTML, replicated in slightly-varying form in dozens of places.”

Don’t get me wrong, I’ve nothing against PHP you can write maintainable PHP code. The trouble is its also easy not to – so depending on the experience of the team it might be prudent to chose a technology that promotes good practice. You will want to extend the application – to deal with new requirements – so architect your application to make it easier to extend.

Scrummaging for unk unks

In 2002 Donald Rumsfeld was ridiculed for his infamous “known unknown” speech. However, despite the ridicule he was in fact quoting an accepted project management theory. That of managing unk-unks.

unk-unks n. especially in engineering, something, such as a problem, that has not been and could not have been imagined or anticipated; an unknown unknown.

Reports that say that something hasn’t happened are always interesting to me, because as we know, there are “known knowns”; there are things we know we know. We also know there are “known unknowns”; that is to say we know there are some things we do not know. But there are also “unknown unknowns” — the ones we don’t know we don’t know.’

When NASA were working on the Apollo programme they realized that, despite their considerable expertise, they had a lot of new problems to solve – including some problems that hadn’t been identified. And because they knew that once the astronauts were in space/on the moon it would be too late to discover they hadn’t thought of something NASA coined the term unk-unk. The engineers then focused on identifying all the unk-unks and solving them before blast off.

With most software projects you are also faced with unk-unks. After all if you’re not solving a new problem why are you writing code? So what should you do about it?

Well you really only have two options. Either, you hedged against by developing on parallel lines and hope that one works out. This is the approach many designers take – they prototype lots of ideas knowing that only one will be picked, and no one can predict which one before all have been seen. Alternatively you can manage the unk unks as they emerge, by changing the approach in response to new information.

If you want to take this second approach then the team needs to be empowered to solve problems en route (rather than only being allowed to build to an existing design and plan). However, in my experience the senior management and/or product owner must also be intimately involved in the project, have sufficient knowledge to check the plausibility of changes, and be supportive of the mid-course corrections.

The project also need to be managed and structured to allow this to happen. And this means the project needs adopt an adaptive, agile methodology, such as Scrum. So the project team are encouraged to solve problems in little steps as they arise and adapt as they learn more.

Wabi-Sabi – Design to age

We’ve recent returned home from camping in France, along with a fairly large number of caravaners. I mention this because I noticed that caravans and campervans don’t appear to be designed to age gracefully. I’m not talking about whether or not they are ‘design classics’ but rather that they aren’t designed to easily allow user modification nor to keep their looks as they age and pick up the inevitable scratches, nicks and general wear and tear. They appear to have only been conceived up to the point of sale – working on the assumption that they will remain perfect forever – with no apparent thought for how they will be affected by use. That’s just silly. Stuff should be designed to take into account general wear and tear.

Tiles

The Japanese have a useful concept to encapsulate this idea – Wabi-sabi (侘寂). Wabi-sabi acknowledges three core principles: nothing lasts, nothing is finished, and nothing is perfect. In other words aesthetics is founded in the acceptance of transience – that beauty is in the imperfect, impermanent, and incomplete.

It seems to me that the same ideas should be applied to our technology products. Too many of our technology products are designed like hi-tech caravans. Lovely and shinny at the point of sale (not that caravans are lovely, but you know what I mean!) – but they don’t stand up well to wear and tear, and they don’t let users easily modify the system to work the way they want to work. They appear to have been designed without thought for how they will age nor how their users might wish to hack the system – to womble – to use the system in unexpected ways.

For software products this means designing small components loosely joined together by a lightweight framework, that support open standards, protocols and APIs; with interfaces that are simple yet flexible enough for users to create their own journeys and workflow.

But Wabi-sabi also tell us something about how to build software. The acceptance of a state of ongoing imperfection implies that attempting to design a solution upfront will be doomed to failure. Instead we should embrace the idea of the perpetual beta, evaluate what works in the real world and modifying the system as a result.

Photo: Stone Tiles, by abundanceguy. Used under licence.

Scaling up agile projects

Agile methods don’t scale, if you are embarking on a big serious project you had better adopt a ‘heavyweight’ method. That way you have certainty, control over your resources and an agreed plan. Right? Wrong.

Agile methods promote using small teams because it improves communications. With improved communications the team members are more likely to understand the problem they are being asked to solve and therefore the proposed solution. Also when the project requirements change the team can more easily adjust. But what about large projects? How can small teams can deliver big projects?

The answer? Divide the big project into small independent projects. Nothing too shocking about that. But there is still a question about how to organise the work and structure the project. Earlier in the year I attended Spa 2007 where Dave Thomas outlined one approach.

The objective is to let projects have freedom to create the solutions to specific problems but to coordinate activity between projects. The approach is to structure the work into four phases:

  • Envisioning;
  • Definition;
  • Development and;
  • Release.

Large Scale Projects

Envisioning

The envisioning team is tasked not with capturing requirements but rather with understanding the problem to be solved, defining the solution and finally building an executable prototype. The work is managed with Scrum but because the purpose is to build a prototype that meets the users’ needs the team works very closely with end users developing their solutions iteratively: adding fidelity and refining the design at each iteration.

If appropriate two envisioning teams can run in parallel developing ‘competitive’ ideas to explore different options.

Definition

The Envisioning Team provides a clear definition of what needs to be built; the Definition team designs how to build it. To do this the Definition Team is tasked with structuring the work and defining the initial Product Backlog. The aim is to reduce dependencies: not just between components, but between the overall requirements and the component designs. The work then is structured into a series of discreet teams:

  • Feature teams (user facing services)
  • Component Teams (core reusable components)
  • Release Engineering (continuous integration and QA)

Feature teams can be responsible for no more than defining the unit tests for component sets – or integrating components with service specific code to provide specific functionality and/or glue sets of components together.

The Release Engineering team is responsible for ensuring that all the components and features work and conform to the agree interoperability framework. They therefore take each release and ensure the ‘system’ works, rather than trying to configure the system at the end of the project.

NB Components can be built out of phase with the rest of the project.

Development

The first job for each of the development teams is to review the product backlog to ensure that the project estimates are OK and the scope is understood.

If any product backlog item can’t be delivered or if there are any project dependencies these get escalated to the ‘risk backlog’ – which is owned by the Product Owner. Its the Product Owners responsibility to clear the Risk Backlog and not allow the individual projects to be effected.

Release

Code is packaged, documentation written, marketing etc.

And of course throughout this whole process everything is kept in the code: code, requirements, risks, people, estimates…

Fall forward quickly

DilbertOne of the problems with the waterfall model is that it forces your client into think through all possibilities at the beginning of the project – and then commit to those requirements and designs or face the potential pain and cost of change control. Unsurprisingly most clients don’t like this situation. But then nor do the development teams, most of whom would rather adopt a more agile approach.

There are plenty of reasons why we end up in this situation. The process might have been foisted upon a project because its seen, by the company, as a way to maintain control over the project costs, scope and delivery time frames; even if there is evidence to the contrary. But I also think it happens because people get use to delivering projects using the waterfall model. So much so that people don’t really understand what it means to work in a more Agile fashion. They don’t really believe that results will be delivered throughout the course of the project, nor that the team should learn what works and doesn’t, and throw away the later.

Instead clients believe that they must ‘get it right’ during the requirements and design phases of the project because they fear that this represents their only chance to influence the final product. Clients therefore tend to throw all their ideas into the pot before the project team sets off on a development death march. It is difficult to get clients to trust that there really is a different, better approach.

It seems to me that to get out of this conundrum you need to demonstrate the approach, to fall forward quickly and deliver tangible outputs en route. This means breaking your project up into independent partitions each with a deliverable every six weeks or so and accept that if you make mistakes early you will get to the solution more quickly.

But how to start? It can help to find an ally and agree a bit of guerilla development to demonstrate the approach to the wider team or client. Just be careful that the entire project isn’t delivered that way. Or when gathering requirements – spend your time in situ and produce wireframes to illustrate the product so that when your done you can build an executable prototype rather than a requirements catalogue. That way you can demonstrate how you plan to proceed.

Then keep moving forward maintaining your deliver timetable. Remain flexible if you get stuck and risk not making your milestone: descope that feature, hit your milestone and put it into the next sprint.

Osmotic communication – keeping the whole company in touch

At the FOWA conference Matt and Anil, from Last.FM spoke about their use of IRC as an internal communication channel to improve their comms across the whole company.

Communication

For those that aren’t familiar with IRC its short for Internet Relay Chat and is a real-time Internet chat protocol designed for group (many-to-many) communication.

One of the things that is interesting about IRC is that it supports automated clients or bots that can be used (queried) to provide specific information via the chat window. So for example, at Last.FM they have a bot called IRCcat (now released under an open source license) that posts to the IRC channel when stuff happens within the development environment – so for example when code is committed to Subversion or a bug closed in Trac these events becomes part of the chat conversation; or it can be used to hand off commands from IRC to another program such as a shell script.

Because everyone at Last.FM has IRC running everyone is aware of what is happening across the company. For example, everyone knows when a bug is fixed, when a new one is entered, when the application throws an exception, or when new code is committed; it also means you can ask the whole company a question, their opinion, let them know the latest news.

This doesn’t (or shouldn’t) mean that all communication is carried out online but it does mean that everyone can keep up-to-date with very little overhead and that this happens passively, by osmosis, rather than relying on a separate (active) reporting process. This is of course good because it means people can focus on their job and data flows out as a by-product; it also means that there’s less (possibly no) opportunity for someone to fudge the data – to hide what is really happening, it’s all out there and transparent.

Now my understanding is that the Last.FM folk’s IRCcat use is restricted to reporting on code and tickets via Subversion and Trac (apologies if I’m wrong).

At SPA Dave Thomas outlined an interesting idea – version everything. Put backlogs, risk and issue logs, time estimates, burn-down charts all into your configuration management tools. Use a wiki or IDE to enter data (tied into your configuration management tools) and tie this into the code (ref); and use the data to automatically generate project reports. The benefits are again obvious: transparency and automation of project reporting. If someone decided to reprioritise the product backlog or reassign work everyone knows when it happened and who did it, and everyone has access to the burn-down charts so everyone knows if the project is on target or not.

I think it would be really interesting to tie these two ideas together: store all your project artefacts in your configuration management tools (entered via a wiki and/or IDE) and automatically generate reports from it; but also provide a live, real-time commentary on the project by hooking in IRC.

Photo: Communications, by assbach. Used under licence.