We Built a Site Just to Throw it Away

We’ve just completed the ‘alpha’ phase of the project to redevelop newzealand.govt.nz by creating a prototype website. Prototyping is a method used to get feedback from users about future designs. In recent times, many governments around the world have used prototyping as a way of increasing the user-centred focus of the site they’re building.

We built our prototype site knowing we’d throw it away to make something that was better. This is kind of scary stuff, and not how most of the project team have been used to working, but it made us focus on what we needed to learn, instead of rushing to build a finished product.

We kept the prototype simple and did just enough to give us a site we could test. To get the site ready we:

  • published about 150 pages of content, across a range of topics
  • created some landing pages to help people navigate the site
  • installed a local search engine
  • left the site deliberately looking a bit rough and the design wasn’t polished

The team worked on the prototype site between September and the end of January. We actually delivered two prototypes, each one with a slightly different layout. Both versions of the prototype were finished on schedule too.

We spent several days in January and February testing the prototype with real New Zealanders. During the testing we made some tweaks, changed some content, modified the design for a few key templates and tested again. We now have pages and pages of research notes and hours of video to digest.

Once again, we worked with Optimal Usability. They helped up recruit test participants, and they ran the test sessions.

Here are the main documents that cover this round of research:

  1. The testing scenarios and test script (PDF 207KB) (WORD 76KB)
  2. The user testing report (PDF 6.69MB) (DOC 8MB)
  3. Implications for ‘beta’ (PDF 80KB) (DOC 92KB)

Sometimes Putting Yourself in Your User’s Shoes Is an Uncomfortable Experience

Watching user testing can be a very humbling experience and sometimes it can make you really uncomfortable too.

During our most recent testing, we asked people to seek out information relating to various situations like “having trouble paying your rent,” “find out if you can get out of jury service,” and more common tasks like “find out where you can pay a speeding fine”. Several participants didn’t always feel they could easily complete the task we had set them.

People think about tasks in different ways, and their different life experiences influence the language they use. Watching what people did on the site pointed to a few problems we could easily fix. The labels for some of the links in the prototype content didn’t quite make sense, contact details for services were still hard to find. I had expected we’d see these sorts of problems, but I was also pleased to see that our overall approach seemed to be working.

Then something happened during the testing that I didn’t expect. Some of the people we tested said it was their fault that they couldn’t find the right content, that they were not good enough using computers, and worse, that it was just something they had to live with. People were apologising to us for not being able to finish some of the tasks. You could see their self-confidence plummet, and this was just a test, they should have been blaming the site for making the test hard to finish, but they were blaming themselves.

A group of us were observing the tests from another room, and it was hard to watch. I thought to myself, “This is not acceptable. No one should feel like this.”

Thankfully, most of the users we tested felt more empowered, and even when they didn’t complete their tasks easily they thought the site would help them deal with “government stuff” more easily. Several users said once they got more familiar with the site it would be exactly what they needed. Here’s what a few of the users had to say:

The consistent layout on each page was good.

—User 1

The info that was needed was right there, so there was no need to search further.

—User 7

I like the cleaner layout compared to some of the sites it links through to.

—User 4

The Lessons We’ve Learned Have Influenced Our ‘Beta’ Plans

We received great feedback on a number of key features and we observed consistent patterns in user behaviour, so if we focus on these areas first, we should end up with a more usable website when the site goes live.

We will need to continue development and adding new features after the site is available to the public, launching the site before it’s finished just helps us collect more feedback along the way.

A variety of themes emerged from the user testing, I’ll run through the highlights here. If you want more detail be sure to download the full set of research documents.

Keep Search as Simple as Possible

The ‘alpha’ site enabled users to search both published content and to search the whole of the government domain. 100% of users we tested said they would start their search for information using Google. Most of them also said they would go back to Google if they couldn’t find something. Local search was useful but only to look for content on the site they were already on.

For ‘beta’: The search engine will only include ‘beta’ content.

Make Content Easy to Find Using Different Pathways

On ‘alpha’ we implemented:

  • automatic suggestions on search queries
  • featured links on the homepage and topic pages
  • content tags
  • a ‘fat footer’ across the site
  • filters that could be used on search results

For ‘beta’: We’re going to keep all of these ‘alpha’ features and focus on fine-tuning search results to make them more relevant.

Ensure Content Supports a Complete User Journey

Users responded really well to pages that summarised the full-range of services that were available around a particular topic, but they still struggled a bit when looking for contact information.

For ‘beta’: We’re calling these ‘sign-post pages’. Linking together services around a specific task is the biggest opportunity for the ‘beta’ site. This is content that government doesn’t really provide now, especially where the full range of services are provided by multiple departments.

We also need to do some more work on how contact details will be displayed. We’ve got some ideas, but this is an area that will need some more testing. Making the ‘beta’ live to the public is probably the best way to discover what users really need. We don’t need to solve these tricky design questions upfront.

Create the Consistent Experience Users Expect

The ‘alpha’ was, as one user put it, A bit messy, and that was fine with us. Showing users something that wasn’t quite finished helped them feel like they could give us ideas on how to make it better. They did, however, like the consistency of the design. It didn’t matter which task they were completing, the basic layout and structure was the same. By the time users had completed two or three testing tasks they knew how the site worked and later tasks were easier to complete.

Each time they were linked out to another website, the experience wasn’t so smooth. In many cases, the inconsistency in the designs slowed users down as they had to ‘relearn’ how each worked. Content on some of the sites forced users to jump back and forth in and out of pages, some of the sites used layouts that were so unique and unexpected they blocked users from completing their task completely.

For ‘beta’: The site will be a publicly available site so it has to look good. We’ve already started work on the first set of templates and we’re experimenting with content from different topics to see if we can create the consistent experience users expect. Early feedback from people who’ve seen the beta mock-ups is, The pages look stunning. I can’t wait to show you more about what we’re building, but that will have to wait for another blog post.

Come on the Journey With Us

We’ve got a busy few months ahead of us here at Internal Affairs. There’s content, designs, templates, and systems to build and test. There’ll be lots of testing to do, and not just technical testing. We’ve got plenty of opportunities planned to show off our ‘beta’ site to users and get their feedback, and we’ll continue to share the lessons we learn along the journey.

Have you got any user research you can share? Let me know and I can help you get it published here on the Web Toolkit.

We’ll publish a sneak preview of the ‘beta’ design as soon as we’ve got something ready to share, but until then, if you’ve got any questions about ‘beta’ feel free to leave a comment.

7 comments

  1. Comment #1. Charlotte:

    Brilliant! Thanks Nathan.

  2. Comment #2. George:

    Great work, I expect to see more of this across more projects in government in the future =)

  3. Comment #3. Nathan Wall

    Thanks George. It’s been great to see how the project team has embraced the “open” approach, it’s become one of our core project principles. We’re looking forward to sharing all sorts of material that the team creates over the next few months, code, designs, style guides, as well as more research and test results. I’m looking forward to the feedback and the conversations we’re going to have with users, it can only help us create a better product.

  4. Comment #4. lesley hanes:

    Thanks very much for sharing your approach and what you learnt – it’s really helpful.

  5. Comment #5. Nathan Wall

    Thanks Lesley, that’s great to hear! Stay tuned for more updates as the project progresses.

  6. Comment #6. Nishant:

    Thanks for sharing your findings.. it is really useful. What’s next now?
    Prototype (PoC) mostly goes to Prod as it is (seen from my past experience), I hope this won’t happen here.
    Looking forward to see more coming across in future.

  7. Comment #7. Nathan Wall

    Hi Nishant, right now we’re working through all the things we’ve learned from doing the alpha version of the site and making sure that the mockup designs we have for beta are addressing the issues we already know about. Going into “production” is still some time away.

    We will be releasing the beta site publicly in a few months, this still won’t be a finished site. The plan then is to run the beta for at least a few months and get feedback from people, then we can adapt and iterate the design and content where needed before going to “production” later in the year.

Navigate Posts