User testing beta.govt.nz — an optimal experience

Optimal Experience have just completed the third and final round of user testing on the beta site. Their findings reassured us that we’re on the right track.

Why test a beta site?

The industry feedback we’ve received is invaluable and we’ve used it to refine a number of aspects of the site — thank you. Beta.govt.nz is designed for our end users — ordinary New Zealanders — and it’s their needs we’ve been striving to meet. User testing works: it addresses functionality, interface design and content. We’re learning more all the time as we integrate our findings into our ongoing development.

We were looking for:

  • task completion time
  • an increase in number of people following the correct path
  • search functionality — ease of use
  • observational feedback on the site overall.

The results: some great, some good … still more to do

The report has given the team reason to yell “yay”, it’s earned a few pats on the back, and given us a few reminders that there’s still more to do.

The top line of the report included:

  • 22 positive findings
  • 2 severe issues that require immediate attention
  • 25 major issues requiring urgent attention
  • 16 minor or cosmetic issues which are less pressing
  • 24 observations which are not problematic but important to note as well
  • 12 key findings and recommendations.

A round of applause please…

Our “boom” moment this round came from our SUS result.  SUS stands for "system usability scale" and is the score that measures the usability of a site. It takes into account:

  • effectiveness — can users successfully achieve their objectives
  • efficiency — how much effort and resource is expended in achieving those objectives
  • satisfaction — was the experience satisfactory.

A SUS score greater than 68% is considered above average.  We scored 70.3% this round, an improvement on round 2’s score of 68.6%.

We were also happy that:

  • users liked the website purpose: The mother ship of all government departments. Definitely use it — would go on and learn things even though I didn't need to use it.
  • content was easy to understand: It's right there. Bullet points, short sentences — done!
  • content highlights also included success with relevant search results (a hard one to achieve when users’ reference point is Google), relevant  search result descriptions and helpful headings and synopses.
  • navigation highlighted users taking the most common paths and easily navigating homepage categories. The high use of popular pages also had us congratulating each other.

Issues already ticked off

We’re constantly iterating, so over a third of the issues raised from round 2 had already been completed by round 3.  Developing in Agile allows our team to be exactly that, which meant we already had a plan of attack to fix:

  • visual design: some users noted the website looked “plain”.  Our latest release incorporates an “NZ” feel with a change in colour, layout and (soon to be included) images.
  • style/layout: refined layout design now includes a prominence of popular pages (which are dynamic) listed on the homepage, plus hub pages now feature the most relevant pages hosted within a hub.
  • external links not leading where expected: link styles were not added prior to round 3 testing.  Design alterations are underway and will be completed for production release.
  • search/filter improvements: design, help text and content have been updated to assist with this.

What to expect when Govt.nz goes live

Juggling priorities and resources leaves some issues lower down the “to do” list — fortunately we'd pre-empted many of the issues and the work is already underway to resolve them (e.g. search improvements).

Current developments planned for go live include:

  • further updates on site search (currently “severe” so this is our highest priority)
  • refinement on Government A–Z listings and search through filter options
  • creating a quicker path for users “to complain about government”.

Talk to us — we love to share

Feedback is valuable, from whatever source.  We're looking forward to getting feedback from users once the site is fully live — we know this will give us more things to look at, and the site will get even better.

We use open source code and our API is public because we believe sharing is important.  In that vein, here's the full report for our third round of user testing. Have a read, let us know what you think, even challenge us — we’re up for it. Just remember that sharing is a two way street — so publish your own reports so we can all keep getting better at what we do.

4 comments

  1. Comment #1. Danielle:

    It’s great the report is accessible, any way we could have a PDF or word version for the people who prefer to read offline?

  2. Comment #2. Andrea Key

    Hi Danielle – thanks for accessing the report. We’ve added a PDF version – look for the two links at the end of the online report. Look forward to hearing what you think of our progress.

  3. Comment #3. ed strafford:

    kia ora,

    fascinating data, very beguiling – but I need to get back to my day job!

    Before I do, a quick query …

    I have only skimmed the graphs -so i may have missed the info i needed to address my query – if so apols.

    Just wondering how you/Optimal accomodated the confounding effect of the age distribution of Māori respondents in your analysis of internet use by ethnic groups.

    By this I mean: i) it seems the Maōri repsondents are atypically young compared to other ethnic groups ii) young folks are shown to be more heavy internet users.

    iii) the ethnic difference graph for internet use shows Maōri as heavy users, but is this a function of ethnicity or (as seems likely) one that is more attributable to the age distribution of your achieved sample for Māori?

    cheers, Ed

  4. Comment #4. Nathan Wall

    Hi Ed, thanks for the question.

    I agree the data is fascinating and having the chance to watch a lot of the testing sessions reminded me that the interactions people have with government are unique – this means we’ve got quite a challenge finding the “right” way to provide information to them.

    To answer your question, we really don’t have enough data yet to draw that kind of conclusion. As we do more research and testing with users, we will be able to compare results over time.

    When selecting participants for the testing, we made sure that the overall representation of ethnic groups loosely matched the most recent census data. Applying a specific range of age groups within the same selection criteria would have made the recruitment of participants more complex though.

    It is possible the younger age range of our Māori participants makes that group show up as being heavy internet users, but it certainly wasn’t the case for all of the people we spoke to. One of our participants was a grandmother with a large extended family and she told us she spent a lot of time online looking for things for her grandkids, finding bargains and presents to buy on online auctions sites and helping out with homework. She’d recently completed a computer course and to her the Internet is a whole new exciting place to explore.

    Are you aware of any other research that could present another perspective?

Navigate Posts