beta.govt.nz — User research (Round 3)
About this report
This is a report completed for the beta.govt.nz site as it was in early 2014. Since this research was completed we've already made some changes to the beta site, so some of the recommendations in this report have already been addressed.
The bulk of this report was prepared by Optimal Experience who conducted the research on our behalf. At various places in the report you will see comment boxes titled "What we think" or "What we're doing". These comments are the views of the project team not Optimal Experience and have been added to the report.
- Executive Summary
- Scores and Ratings
- Task Related Findings
- General Findings
Between February and April 2014, Optimal Experience undertook six sessions of user testing with 30 participants in six different locations throughout New Zealand. A beta version of the govt.nz website was tested.
This project followed earlier rounds of user testing in October 2013 for the same website.
User Testing Process
Objectives of the Study
The objectives of this project were to find out:
- What search terms do users use in Google and how does landing, following a Google search result, on a deep content page effect the users’ performance?
- When and why do users use the local site search, what search terms do they use and what problems do they encounter?
- What pathways do users take when approaching a task?
- Do users use the ‘Topics’ (aka ‘Browse.govt.nz’) and ‘Contact government’ (aka ‘Government A-Z’) sections and do they understand how they work?
- Can users easily find information about Departments and Agencies?
- Do users have problems with the different information hubs on the site?
- Can users find contact details easily?
- Do users notice when content links to other agency websites and are they comfortable with that?
- What do participants make of the ‘no frills’ design?
The user tests were carried out in six different locations over a 3-month period:
- Auckland on Monday, 17th February
- Manukau on Wednesday, 19th February
- Porirua on Thursday, 6th March
- Christchurch on Thursday, 13th March
- Dunedin on Thursday, 3rd April
- Nelson on Tuesday, 8th April
Participants were given several tasks to complete using the beta.govt.nz website. Most participants completed all nine tasks while some, due to time constraints completed less tasks.
Some tasks were slightly altered between the rounds of testing to test different aspects of a task, but remained essentially the same.
There were also some formatting and layout changes to the website itself during this timeframe.
Participants were asked to start some tasks with Google, but otherwise were given the choice between browsing and using the local site search.
As part of the testing session, participants were also given two exercises in order to capture their perceptions of NZ government sites in general, and the beta.govt.nz site in particular.
Usability Findings Summary
The user testing revealed 89 findings which were given a usability rating which indicates the severity of the finding and to which extend this finding will impact on the user experience.
- 22 positive findings,
- 2 severe issues that require immediate attention because they prevent users from completing their tasks, appear frequently or in crucial situations,
- 25 major issues that make it difficult to use the site and should be fixed as soon as possible,
- 16 minor issues that should be fixed if resources are available because they can cause irritation/confusion or occurs rarely,
- 24 observations which are not problematic but important to note as well.
Each minor, major or severe issue is accompanied by a recommendation, how this issue could be solved.
Key Findings and Recommendations
Liked Website purpose.
Most participants understood the purpose of the site and liked that it was a one-stop shop for government information. Most participants said that they would like to use it.
“The mother ship of all government departments.”
“Definitely use it - would go on and learn things even though I didn't need to use it.”
Govt.nz website better than other government websites.
Some participants commented that the govt.nz website was easy to use and made them realise how bad some other government websites were.
“That was fairly easy, it's all straightforward. Most government websites are intimidating.”
Content easy to understand.
Participants found information easy to understand and apprehend.
“It's right there. Bullet points, short sentences - done!”
Would start with Google and take it from there.
Most participants said that they would start looking for information by using Google.
“Anything I want to know I just Google it, grandchildren tell me to.”
Younger participants less patient.
Older participants seemed more willing to explore and try different avenues when they couldn’t find information straight away.
Task success not reflected in SUS.
A high task success rate did not necessarily result in a high System Usability Score (SUS). For example Christchurch had the highest task success rate (93%) but the lowest SUS (63.5%).
Prefer direct contact for help.
Most participants said that they would call somebody or the department in question if they weren’t able to find the information online. Participants who got stuck, usually looked for a number to call or a way to contact the responsible department or agency.
“It’s sometimes easier to call; it saves you clicking. … It’s a habit.”
Most participants found the website plain and unattractive. Though this didn't seem to be a problem for most participants; when asked what they would change about the site, a quarter of
participants said that it should be more appealing.
“Would make it a bit more appealing for the eye - not saying put flowers. It’s not attractive ay. I'd rather have effective and efficient rather than attractive.”
“I wouldn’t enjoy looking at that, there’s no pictures, nothing that screams excitement.”
Consider making the website visually more appealing to improve ‘perceived usability’.
What we're doing
We're already working on the next iteration of the look and feel for Govt.nz which should be live by June this year.
Search often unhelpful.
Multiple search terms and misspellings usually resulted in a ‘No results’ message or in irrelevant search results.
“This doesn't help me at all.”
Make sure the search handles multiple search term queries well and if returning no results, provide suggestions on how to improve the search results.
Cater for common misspellings.
What we're doing
We were still working on fixes and enhancements to the site's search engine when this research was undertaken. We've already made some changes to the way the search engine works, it should now handle multiple search terms better than it has previously. We're also setting up synonyms for common terms and phrases based on actual search keywords used by people visiting beta.govt.nz
Government A-Z needs improvement.
Participants often overlooked the ‘Government A-Z’ section on the homepage. On the ‘Government A-Z’ page they often overlooked the filter and had to scroll the list up and down in order to find the department or agency they were looking for. Participants tried to but weren’t able to filter by subjects.
Consider moving the ‘Government A-Z’ section above the fold to the top part of the homepage.
Consider enabling users to filter the ‘Government A-Z’ page by subject and not only by department name.
What we're doing
The next version of the homepage moves some of the elements around and should make both the Government A-Z and the Topics page easier to find. We're going to track which pathways users follow to measure if the changes have had a positive impact.
Browsing too tedious.
Some participants found the website tedious to navigate and felt that it had too many levels.
“A maze of levels.”
Enable users to complete tasks more quickly, e.g. by providing quicker access to the footer, making search more reliable and making the ‘Topics’ page layout another layout option for the homepage.
Missing content by not scrolling.
Some participants missed important information on the Homepage like 'Government A-Z' or on other pages because they didn't scroll.
“I didn’t scroll down. Usually there is a bunch of un-useful information at the bottom, it would need to be at the top. I didn’t even see that, I scrolled past it like 3 times.”
Consider including page elements that stretch over more than one page length.
Consider removing horizontal lines that reach across the whole page width.
For this round of user testing we tested 30 participants in total in six different locations, i.e. Auckland, Manukau, Porirua, Christchurch, Dunedin and Nelson.
There was an even split between genders and a good spread across the different age groups.
Based on the most recent census data from Statistics New Zealand, we selected participants from different ethnic backgrounds including NZ European, Pacific Islander, Māori and Asian.
When asked what things they used the internet for, most participants responded
- Facebook (73%),
- Doing research (50%),
- Emails (37%),
- TradeMe (33%),
- And watching or downloading videos, movies or TV (33%).
Almost every participant said that they would use Google to find information online.
NZ Government Websites and newzealand.govt.nz Usage
Most participants had used NZ government websites before (93%) but only a third of participants had heard about newzealand.govt.nz
Most thought that the website would have something to do with NZ government or government departments.
“I imagine it’s an overall website that covers all the government departments.”
The NZ government websites most mentioned by participants were:
- Inland Revenue (43%),
- Work and Income (30%),
- Local Council websites (30%),
- Department of Internal Affairs (23%),
- New Zealand Transport Agency (17%),
- StudyLink (17%),
- Library websites (17%),
- And public transport websites (17%).
Most participants rarely visited NZ government websites, i.e. once a month or less and quite often it depended on the participant’s needs or the time of year.
“They are not compelling, you only use it because you have to.”
Most participants said that they would use Google to find government information and services (80%), some said they would prefer to talk to somebody in person either face-to-face (17%) or over the phone (13%).
“I would Google it and hope that something comes up.”
Three quarters of the participants said that they would definitely use the site or use it when they needed to.
“Probably go on tonight and use it, found it really interesting. Lot of things I didn't know about.”
“I will certainly put it in my favourites.”
“I don’t need it I guess, haven’t really got any problems at the moment.”
“I’m going to bookmark this site and use it all the time.”
“I actually like it, it’s really good.”
Only 10% said that they wouldn’t use it.
“I wasn't overly impressed - definitely room for improvement.”
“Not well integrated, information too much all over the place”
“I have to read too much, there was lots of information to decipher first. You would have to get use to the scrolling and knowing the footer and what not.”
In order to gain an insight of users perceptions of government websites in general and the govt.nz website in particular, we asked participants, in two separate exercises, to sort a list of 35
adjectives into two categories; one that included the ones that applied to government websites or the govt.nz website and one for the adjectives that didn’t apply.
NZ Government websites
The words that participants most commonly associated with NZ Government websites were Professional (93%) and Helpful (89%).
Also high in the rankings were Usable (79%), Organised (79%), Reliable (78%), Convenient (78%), Secure (78%), Trustworthy (75%), and Ordinary (71%).
There was also high agreement that NZ Government websites were NOT Fun (96%), NOT Stimulating (89%), NOT Incomprehensible (85%), NOT Patronising (81%), and NOT Inconsistent (79%).
The words that participants most commonly associated with beta.govt.nz were Reliable (74%), Organised (70%) and Trustworthy (70%).
Also high in the rankings were Convenient (68%), Helpful (68%) and Friendly (68%).
There was also high agreement that beta.govt.nz was NOT Impersonal (70%), NOT Slow (67%), NOT Hard to Use (61%), and NOT Intimidating (60%).
Main perception differences
There were some interesting differences between how participants perceived NZ government websites in general and beta.govt.nz in particular:
|Perception||NZ Government websites||beta.govt.nz|
Scores and Ratings
The overall Usability (SUS) score for the govt.nz website was 70.3%, which is higher than average.
A score of 70.3% indicates that people agreed that the website is easy to use. For example, a score of 50% would mean that respondents neither agreed nor disagreed that the website was usable and a score of 100% would mean that everyone strongly agreed that the website is usable.
A score of 68% is considered average. In the book Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 50 SUS scores were analysed from published usability studies that determined that the mean score was 66% and the median as 69%.
SUS by Location
The SUS varied widely between the different locations with the lowest score in Christchurch (63.5%) and the highest score in Dunedin (77.5%).
It is important to note that there is only a modest correlation between the SUS and the users’ task performance. For example in this test, Christchurch had the highest average task success rate (93%) but the lowest SUS (63.5%) while Manukau had the lowest average task success rate (52%) and a higher SUS (70%).
SUS by Ethnicity
The SUS also varied widely between people with different ethnicity with participants of Asian background giving the lowest score (59.5%) and participants of NZ European background giving the highest score (78.3%).
SUS by Age Group
Looking at SUS by age group, the website scored the highest usability score with 30-49 year olds (73.5%), while under 20 year old gave it the lowest score (65%).
SUS by Participants
The SUS score also varied widely between participants with 30% being the lowest score and 97.5% being the highest score given.
More than two thirds of participants gave the website a higher score than 70%.
For all tasks the following measurements were recorded:
Task success rate: Shows how many percent of participants managed to complete the set task successfully within the set parameters.
Task directness: Indicates how directly participants completed a task by showing how many percent of participants managed to complete the tasks through the shortest and most straightforward route. For example, 80% directness means 80% of participants managed to complete the task through the most direct route.
Time taken: Shows how long it took participants to complete a task. The times included in this report represent the average time taken across all participants who completed a task.
Overall, Task 3 – Consumer Rights was the best performing task with the highest task success as well as directness. Only one out of 24 participants who attempted this task, failed to complete it.
Task 8A – Complaining about a government department was the worst performing task. Only 10 out of 26 participants who attempted to complete this task managed to complete it successfully.
The tasks with the highest success rates (Task 2, Task 3, Task 5 and Task 6) were also the ones completed the fastest while the least successful task (Task 7) was also the slowest to be completed.
Comparing the task success rates between the different locations and with the overall success score shows that Manukau performed the worst and Christchurch the best.
‘NZ European’ participants completed the most tasks (80%) and ‘Pacific Islander’ participants the least (66%).
Overall participants over 50 years old (79%), between 20 and 29 year old (75%) and between 30-49 year old (74%) were more successful in completing tasks than participants under 20 years old (61%).
Task related findings
Task 1 – Low Income & Housing Support
Participants were given the following task: “Imagine you’re not working as many hours as you used to and you’re now having difficulty paying your rent. What would you do? Where would you look for help?”
Overall Task 1 received the following scores:
- Task success rate: 87%
- Task directness: 57%
- Average time taken: 1m 43 sec
1 - Most common path for Task 1 (Observation)
More than half of the participants who attempted this task selected the 'Help paying rent and housing costs' in the search results or on the 'Money, benefits and tax' page. Here most participants selected the 'Check what benefits you might get' link to go to the Work and Income website.
“I suppose it’s like an invitation to receive help, it’s telling me what I might get.”
2 - Google search for Task 1 (Observation)
Most participants who used Google for this task used the search term ‘Work and Income’ or ‘rent’.
A third of those selected the 'Don't have enough income' search result option because it was a link to the Work and Income site or because it seemed the most fitting one for the participant’s situation.
Three said they wouldn't select the 'Budgeting and managing your money' option because it wouldn't help them when they needed help immediately.
“Because it [Don’t have enough income] says what I don't have.”
3 - Help from the government (Observation)
A third of participants said they would first talk to their Landlord, ask their family for help, get another job or talk to their bank before seeking help from the government.
Some even said they wouldn’t go to any government department because of pride or because they needed to provided too much private information to get help.
Most thought that Work and Income would be the right government agency to help with this problem; with a third of the participants wanting to go straight to the Work and Income website. Though participants who were students mentioned that they would go to StudyLink first.
“They will want to know everything about you.”
“I'd stop here. I'd do everything I could do before I had to give them my information.”
4 - Homepage category for Task 1 (Positive)
All participants who tried to find the answer to this task by going through the homepage either went to ‘Money, benefits and tax’ or ‘Work and jobs’.
“That’s more relevant to me.” 30-49 year old (Pacific Islander) from Porirua
"I was gonna go for a search but I’m thinking this [Money, benefits and tax] might help."" 50-65 year old (NZ European) from Nelson"
5 - Couldn’t find information on external site (Major)
Participants who didn’t complete this task usually had trouble finding the information they required.
Most said that they would look for a phone number to call the responsible department.
“I feel a bit stuck with that.”
“Would be helpful to see something like 'are you struggling to pay rent, buy food'. Something that acknowledges that you're in the right place, could be so stressed in that situation and panic.”
“I'd probably have to ring someone up - it's a bit complicated.”
Consider providing enough context and information on the site before offering a link to an external website, so users don’t feel the need to go to
Task 2 - Jury Duty
Participants were given the following task: “You’ve been selected to be on a jury and you’d like to try to get out of it. Find out if you can do that and why.”
Overall Task 2 received the following scores:
- Task success rate: 93%
- Task directness: 73%
- Average time taken: 1m 22 sec
6 - Most common path for Task 2 (Observation)
Most participants selected the ‘Crime, law and justice’ category on the homepage and then the ‘Jury Service’ option in the ‘Popular pages’ section.
7 - Search for Task 2 (Observation)
Most participants who used Google or the site search for this task used the search term ‘jury service’ or ‘get out of a jury’.
Most participants that used Google went straight to the Ministry of Justice website.
8 - Homepage category for Task 2 (Positive)
All participants who tried to find the answer to this task by going through the homepage went to ‘Crime, law and justice’.
9 - Information easy to understand (Positive)
Participants commented that the information was easy to understand.
“It's just simple how it's worded.”
10 - Missed information because didn’t scroll (Major)
Some participants missed the ‘Read 'If you can't do jury service' section or the ‘Apply to be excused from jury service’ link because they didn't scroll all the way to the bottom of the 'Jury Service' page.
“Won't blame the site, I didn't read further down.”
“Would have been easier if I took my time, scrolled to the bottom of the page - instead of just read the top bits.”
Consider displaying a page index or page summary at the top of the page with links to key sections.
What we think
We've seen this work pretty well on hub pages that show 'popular pages' a featured section and links to other sub categories. This is probably worth trying. We'll set it up so that the template inserts the links on the page automatically to save our editors from having to create and change links as content changes. For this particular page we've also split it into 2 different pages. The old version of the page was trying to address 2 completely different user needs on a single page - being on a jury, and getting out of jury service.
11 - Information not applicable to user’s situation (Minor)
Two participants were unsure if they could be excused from jury service because they were studying and not working.
Consider including information or references that specifically relate to students.
What we're doing
We will put this content back through our review and editing process, information for students might not be the only things that's missing from the page.
12 - Popular pages too specific (Observation)
Four participants didn't select 'Jury Service' from 'Popular pages' on the 'Crime, Law and Justice' page but the 'Courts and Jury service' link further down, because the ‘Jury Service’ link looked too specific.
“Maybe that popular pages kinda threw me a bit. ...Not sure why, sometimes you think some of these sites the simplest one isn’t the one.”
Task 3 – Consumer Rights
Participants were given the following task: “You bought a computer from Harvey Smith 18 months ago, but it now keeps crashing. You take it back to the shop but they tell you it’s not covered since it’s more than a year old and want to charge you $800 to fix it. You don’t think that’s fair, is there anything you can
Overall Task 3 received the following scores:
- Task success rate: 97%
- Task directness:83%
- Average time taken: 1m 46 sec
13 - Most common path for Task 3 (Observation)
Most participants selected the ‘Consumer rights’ category on the homepage and then the 'Your rights even if you don't have a warranty' option in the ‘Popular pages’
section or the 'Consumer disputes and complaints' option and then the 'Your rights even if you don't have a warranty' link.
Some also selected the ‘Complain about faulty goods’ or the ‘Your rights when making a customer complaint’ options.
14 - Homepage category for Task 3 (Positive)
Most participants who tried to find the answer to this task went straight to ‘Consumer rights’.
Some selected 'Consumer rights' after going to a different section, i.e. 'Internet, Media and Communication', 'Government and politics', 'Crime, Law and Justice', but realised that they were in the wrong place, after looking at the page.
15 - Information easy to understand (Positive)
Most participants found this task very easy to complete and the content easy to understand.
“This is easier, a lot easier.”
“Good, it gave you all the avenues you could follow. And it didn’t take long to find that.”
16 - Options too similar (Minor)
Some participants were unsure which option on the 'Consumer disputes and complaints' page was the right one to choose because they sounded quite similar, e.g. ‘Complain about faulty goods’, ‘Your rights even if you don't have a warranty’ and ‘Your rights when making a consumer complaint’.
“I’m playing a guessing game?”
Consider if all pages are necessary or if they could be combined.
Task 4 - Contact Phone Number
Participants were given the following task: “You want to contact the [government department]. Find a contact phone.”
Most participants were asked to find the phone number for the Ministry of Fisheries. Other government departments included Work and Income and Ministry of Justice.
Overall Task 4 received the following scores:
- Task success rate:83%
- Task directness: 25%
- Average time taken: 1m 57 sec
17 - Unsure about department responsibilities (Observation)
Some participants who were asked to find a contact phone number to enquire about fish catch limits were unsure who was responsible for those.
Some thought the Department of Conservation was responsible and went to the ‘Environment and climate’ section.
18 - Contact information in footer (Positive)
Some participants found the contact information through the footer, either by clicking on the ‘Contact government’, ‘Government A-Z’ or the agency name.
19 - Unfamiliar with new ministry structure (Observation)
Half of the participants didn’t know that Ministry of Fisheries is now part of Ministry for Primary Industries.
20 - Helpful synopsis (Positive)
Almost all participants who used the site search to find the Ministry of Fisheries selected 'Ministry for Primary Industries' from the search results list.
Though some participants were slightly confused when the search didn't return 'Ministry of Fisheries' as a result the synopsis underneath the heading helped them to understand that the 'Ministry for Primary Industries' search result was the right one.
“Oh! Found it. Oh okay, now it looks like MPI is covering Fisheries.”
21 - Right panel obvious (Positive)
Participants who found the agency page saw the phone number in the right panel.
22 - ‘Hidden’ contact details (Major)
A third of participants who attempted this task selected a category on the homepage that seemed related to the agency they were looking for, e.g. 'Community, arts and
sport' or 'Environment and climate' for 'Ministry of fisheries', 'Crime, law and justice' for Ministry of Justice or ‘Money, benefits and tax’ for Work and Income.
Most gave up on the second or third level (Information hub Level 2 and 3) because they couldn’t see any contact information on these pages.
“I’m lost now.”
Consider providing prominent links to subject-related contact information on every page, e.g. in the right panel.
23 - Site search for Task 4 (Observation)
Most participants who attempted this task by using the site search used the name of the agency, e.g. ‘Ministry of fisheries’ or the subject about which they wanted to contact the agency, e.g. ‘fishing limits’.
Task 5 – Name Change
Participants were given the following task: “Imagine that a female friend of yours has changed her name/has got married and has changed her name and
now needs to change the details on her driver license. Find out how to do this.”
Overall Task 5 received the following scores:
- Task success rate:97%
- Task directness: 64%
- Average time taken: 1m 13 sec
24 - Most common path for Task 5 (Positive)
Almost two thirds of participants who attempted this task selected 'Driving and transport' on the homepage and then the ‘Driver licences’ and ‘Change your name on your driver licence’ links. All participants who followed this path found the right information.
“That was easier than the last few - saw driver licence on the homepage.”
25 - Browsing by context (Observation)
Almost half of the participants selected the 'Births, deaths and marriages' category from the homepage because they saw changing your name in the context of marriage.
26 - Obvious link on the ‘Change name’ page (Positive)
Most participants who went to the ‘Change name’ page saw and selected the ‘Change name on driver’s license’ link.
“I thought that was quite easy. I don’t usually find things that easily.”
“That was quick. I had a legalised name change in my mind, but I took the easy route. I'll change my name to Phil Dotcom; I’ll earn millions.”
27 - Information easy to understand (Positive)
Most participants found this task very easy to complete and the content easy to understand.
“It clearly says what you need to change the name.”
“Simple. Had to be simple or I wouldn’t have got there.”
28 - Confusing terminology (Minor)
Three participants were confused by the term ‘replace’ in the ‘application for a replacement driver licence’ link.
They didn’t seem to see the heading ‘If you’re changing your name, you have to replace your licence’ because they were concentrating
on the link.
Consider providing an explanation close to the ‘application for a replacement driver licence’ link that explains why users need to replace the driver licence.
Task 6 - School Holidays
Participants were given the following task: “You've just seen a really good holiday deal online for 5 days on the Gold Coast. You want to book it as it will be gone soon but need to check if your kids will be in school. Where would you find this information?” or “Find out when the next school holidays start.”
Overall Task 6 received the following scores:
- Task success rate:91%
- Task directness: 77%
- Average time taken: 1m 14 sec
29 - Most common path for Task 6 (Positive)
Three quarters of all participants who attempted this task selected the ‘Education and training’ category on the homepage and the ‘School terms 2014’ option either in the ‘Popular pages’ or the ‘Schools and colleges’ section straight away.
“Really easy, couple of clicks away and right there. Might even have it set as my homepage/bookmark. Something I'd refer to quite a bit.”
“Oh, that was quick. That was just so easy.”
“So far I can tell you it’s very, very easy, and the pages complimented each other.”
30 - Synopsis not applicable to user’s situation (Minor)
Three participants hesitated to select the ‘Education and training’ category on the homepage because they synopsis underneath the heading didn’t cover their individual
situation, e.g. kids in primary school.
Consider providing a more detailed synopsis on mouse over.
What we think
This is a real challenge. We have to balance the need to present enough information to users to help them choose the pathways through the site, but we need to do it in the most accessible way. Showing additional information in a tooltip that displays on mouse over won't work for users who rely on keyboards or other devices to navigate the web, and there will be inconsistencies in the way assistive technologies interpret and present the information. We think the first place to start is to review the summaries under each category and see if we can improve them. Feedback from users here is really welcome.
Task 7 – Travel Overseas
Participants were given the following task: “(You’re getting ready to go on Holiday to the Gold Coast with your family.) You receive accommodation supplement (and childcare subsidy). You're only going away for a week. Do you have to tell anyone in Government you're going?”
Overall Task 7 received the following scores:
- Task success rate: 33%
- Task directness: 21%
- Average time taken: 1m 42 sec
31 - Correct path for task 7 ignored (Major)
Only a quarter of participants who attempted this task went to the ‘Before you travel’ page in the ‘Passport, travel and tourism’ section.
Some participants said that they wouldn’t go to ‘Passports, travel and tourism’ because the synopsis didn’t say anything about benefits.
“I'm going to look in the benefits [section] - not thinking holidaying and passports. It should have been under the benefits section.”
“Yeaaah, it’s a bit out of left field. Na, I wouldn’t have thought it would have to do with the benefit."" 50-65 year old (NZ European) from Nelson
Provide a link to the 'Before you go' page from the 'Money, benefits and tax' section.
What we're doing
The information people needed was actually already in the 'Money, benefits and tax' section but we think it might not have been prominent enough. We'll put some of the pages in this section back through our content review process and see if we can improve them. We might need to split some pages where they are currently trying to address too many user needs in one place.
32 - Found required information (Observation)
Most participants who went to the ‘Before you travel’ page found the information about benefits.
33 - Search by benefit type unsuccessful (Major)
Most participants were unsure where to get started and used the site search searching by benefit type, e.g. accommodation supplement but didn’t get any or only irrelevant results.
What we think
We've kept the content on the beta site pretty thin, and this has meant we don't actually list the specific benefits and payments people may be able to get from agencies like Work and Income. Instead, we linked users to the 'Find out what you might get' tool but this means we've ignored that group of users that has some familiarity with what benefits are available, or at least what they used to be called. We'll take another look at the content affected by this and make some improvements.
34 - Concentrating on the benefits aspect (Major)
Participants who approached this task from a benefit and not a travel point of view, usually didn't manage to complete this task successfully.
A third of participants selected the ‘Money, benefits and tax’ category on the homepage.
One participant who used the search term 'travel overseas on benefit' didn't select the 'Before you travel' search result but went to 'Benefits from Work and Income' page.
“I was focusing on benefits, not in terms of travel.”
Ensure that users can complete this task when approaching it from the 'benefit' angle.
Consider including a link to the 'Before you travel' page in the 'Money, benefits and tax' section.
35 - Task not completed or abandoned (Observation)
Four participants gave up on this task.
Most participants who didn’t complete the task said they would call the responsible department, e.g. Work and Income.
“It's just too hard - if I was at home I'd cry, or call someone.”
“Usually I jump on the phone and use their 0800 number.”
36 - Misleading homepage category (Minor)
Two participants selected 'Work and Jobs' on the homepage because they were looking for information from Work and Income and thought Work
and Income would be under there.
Consider renaming this category to ‘Jobs and work’.
What we think
Having seen users work through this task, it's now really easy to see why some people would confuse 'Work and Income' the department, with 'Work and jobs' the topic. We will make this change before the beta period finishes and the site goes fully live.
Task 8A - Complain about Government Agency
Participants were given the following task: “You have been dealing with [government agency]. It's been a nightmare and you're really unhappy. You already complained to the responsible government agency/department without success. Find out if there is another way you can complain about this agency/department or an impartial party to complain to.”
Government agencies we asked participants to complain about included IRD, ACC, StudyLink and Work and Income.
Overall Task 8A received the following scores:
- Task success rate: 34%
- Task directness: 10%
- Average time taken: 3m 01 sec
37 - Most common path for Task 8A unsuccessful (Major)
More than half of the participants expected to find the answer to this task under ‘Consumer rights’ but couldn’t find it there.
Some participants went to this section because it included the term 'complaints' in the synopsis under the heading.
Consider providing a link to the ‘Complain about a government agency’ page in this section.
38 - Expected complaint option in the footer (Major)
Some participants looked for complaint section in the footer, because that it's were it was on most government pages.
Consider providing a link to the ‘Complain about a government agency’ page in the footer.
39 - ‘Consumer rights’ page provided good overview (Positive)
Most of the participants who went to the ‘Consumer rights’ section realised that it wasn’t the right section, after viewing the content of the ‘Consumer rights’ page.
40 - Relevant search results (Positive)
In general, participants who used search used the term 'complaint' in combination with the name of the agency they wanted to complain about.
For most, the search returned results including 'Complain to the ombudsman' and 'Complain about a government agency'.
41 - Didn’t know about the ombudsman (Observation)
Half of the participants were unfamiliar with the term ‘ombudsman’ and didn’t know who it was.
Even some of the participants who knew about the ombudsman didn’t associate complaining about a government department with this role straight away.
“Complain to the who?”
“I don’t know, is that a person or a department?”
42 - No clear starting point on the homepage (Major)
Most participants were unsure where to get started on the homepage and were focused on the agency or subject they wanted to complain about rather than the complaints process itself.
Some selected categories which seemed to relate to the agency they wanted to complain about, e.g. ‘Money, tax and benefits’ for IRD, ‘Education and training’ for StudyLink, 'Passport, travel and tourism' for DIA and from there to the agency page.
Some selected 'Crime, law and justice' on the homepage and then 'Legal rights' on the 'Crime, law and justice' page.
“Can't see anything immediately on the homepage about complaints”
Consider including a link to the 'Complain about a government department' section at the bottom of each department page.
What we think
We think we can make information about making a complaint easier to find, but we can also help users by making it easier to find the information about getting help when dealing with government. Adding something to each organisation's page in the 'Government A-Z' section is something we've added to our backlog of work to do.
43 - ‘Parliament and politics’ heading not working (Minor)
Nobody selected the 'Parliament and politics' heading to find the complaints section.
Some participants selected the 'Government and politics' on the homepage and then on of the 'Complaint' links on the 'Government and politics' page.
“Doesn’t make sense to me, or to the common person, it would be more about policy, I’d be looking for advice, complaints.”
Consider renaming the 'Parliament and politics' section to 'Government and politics' or move the 'Complain about a government department' link to the 'Consumer
44 - Task abandoned (Observation)
Almost a quarter of participants gave up on this task.
Most said they would go to the department website to find out how to complain about the department.
“Don't know what it would be under. Would probably just Google it.”
“That was kind of complicated, found that complicated.”
Task 8B - Minister of Government Agency
Participants were given the following task: “Find out who the Minister is who is responsible for [government agency/department].”
Government agencies we asked participants to find out about included IRD, StudyLink, Work and Income, Ministry of Education, Ministry of Fisheries and Ministry of Justice.
Overall Task 8A received the following scores:
- Task success rate:51%
- Task directness: 10%
- Average time taken: 2m 13 sec
45 - Most common path for Task 8B unsuccessful (Major)
Half of the participants selected the ‘Government and politics’ or ‘Parliament and politics’ category on the homepage.
Most then selected the ‘Parliament’ and then the ‘Members of Parliament’ option and ended up on the Parliament website where they didn’t find the information they were looking for
“Don’t think this is the right way.”
Consider providing a list of Members of Parliament on the website.
46 - Unsuccessful search (Severe)
Almost half of the participants who attempted this task used the site search.
Most searched for the term 'Minister' in combination with department name, e.g. 'Minister for ACC' which resulted in a ‘No search results’ message.
Ensure the site search caters for multiple search terms.
Ensure a 'No results found' message includes recommendations and suggestions how to improve the search.
What we're doing
We'll be making some changes to the way the search engine indexes content on the site. We've also made sure the search engine is indexing the names and roles of the Ministers so there should be less no-results pages occurring in the future. We also know we won't have all the content people are looking for, so we'll make it easier to repeat their search on Google when the local search engine doesn't return any results.
47 - Missed information because didn’t scroll (Minor)
Some participants who got to the right page, i.e. the agency page, didn’t scroll the page and missed the information about the Ministers.
Consider displaying a page index or page summary at the top of the page with links to key sections an moving the information about the
Ministers further up the page, so it doesn’t fall below the fold.
48 - Missed ‘Government A-Z’ because didn’t scroll (Major)
Most participants missed the ‘Government A-Z’ or ‘Contact government’ section on the homepage because they didn’t scroll.
More than a third of the participants selected a category heading at the top of homepage which described a subject which the department or agency would be responsible for, in order to find the Minister responsible for it, e.g. ‘Crime, law and justice’ for Minister for Ministry of Justice, ‘Money, benefits and tax’ for Minister responsible for IRD, ‘Education & training’ for Minister of Ministry of Education, 'Benefits from Work and Income' for Work and Income, 'Community, arts and sports' for Department of Conservation.
Consider moving the ’Government A-Z’ section to the top part of the homepage.
49 - Found ‘Government A-Z’ in footer (Positive)
A quarter of participants selected the ‘Government A-Z’ heading in the footer after trying different ways to find the answer to this task.
50 - Task abandoned (Observation)
A quarter of participants gave up on this task.
Most said they would use Google to find the answer to this task.
Website Purpose and Scope
51 - Scope of website unclear (Observation)
Some participants were unsure if some agencies were government or government-related agencies and therefore were unsure if they would find information about this agency on the site, e.g. NZTA, Libraries.
52 - Website unknown (Major)
Most participants didn't know the newzealand.govt.nz site, so purpose of the site wasn't immediately clear.
Some participants didn't select govt.nz from the Google search results, because they were not familiar with the site.
Two participants even thought that the website wasn’t genuine.
“Looks like a hoax website - like when you type in an address wrong.”
“Wouldn’t think it was an actual government website, because of the layout of it.”
Promote the website and educate users about the purpose of the website.
Consider including a link to the 'We're govt.nz' section in tag line, so users can find out more about the govt.nz project and the people behind it.
53 - Good source of information (Positive)
Most participants liked the site and found the information it useful and thought that it would have all the information about NZ government they would need.
Four participants commented that it would be a good site for people who are new to New Zealand.
“Would be great for people just moving here.”
“Has all the government information that consumers like and need to know about.”
“govt.nz it’s like a whole lot of brochures, then its takes you where you need to go."" 50-65 year old (NZ European) from Nelson
“Don't think people would struggle using it.”
“My nanna she would find it quite easy to do this.”
“Doesn't make me as apprehensive. Put's me at ease, makes me feel like I can do it quite easily. Feel like I could do it by myself which is great, got everything anyone could possibly need.”
54 - URL shows it’s a government site (Observation)
Participants commented that the ‘.govt’ part of the URL told them that beta.govt.nz was a government site.
55 - Better than other government websites (Positive)
In general, participants found govt.nz easier than other government department websites.
“Oh yeah, [other government website] is quite a muddly website, text is small and bunched together, different colours all over the place, old fashioned type fonts.”
“[other government website] it’s so squished in.”
“That was fairly easy, it's all straightforward. Most government websites are intimidating. … Four kids, around me all the time when trying to concentrate. Don't like it all crammed up together.”
“The [other government website] has got so much going on. It’s everything, every [service], all the info. This is just cutting to the chase.”
“It was easier than I thought it would be.”
56 - Website looks plain (Major)
Participants thought the website was quite plain, boring and unappealing.
While most didn't seem to have a problem with this, some quoted it as the one thing they would change about the site.
“It’s not too appealing. It’s not really eye-catching, kind of boring.”
“The look is relatively boring, but it is practical.”
“I think that's a bit too plain - it works.”, 20-29 year old (Pacific Islander) from Auckland
“Like info to be clear, don't need fancy stuff all around it.”
“You're looking for information, you don't need to be entertained by it. You can't have Mickey Mouse running across it.”
Consider making the website visually more attractive without adding clutter to improve the perceived usability.
57 - Time consuming and long winded (Major)
Some participants found the site too tedious and time consuming to get through.
Two participants relied primarily on the footer to navigate the site, once they had found it.
“It seems to be within 3 or 4 clicks to find something. I’m not a very fast reader, so my eyes are usually flying all around the page.”
Consider providing shortcuts to frequently used information.
Ensure that users can use the site search to quickly find information.
58 - Browser back button commonly used (Observation)
Most participants used the Browser Back button to navigate the site.
“It's something I’ve always done.”
59 - Homepage link not obvious enough (Minor)
More than a third of participants used the 'Home' link in the bread crumb trail or the site logo to return to the homepage, but some participants didn’t see the bread crumbs or didn’t see the ‘beta.govt.nz’ text as a logo.
One participant tried to return to the homepage and when using the browser back button skipped past the homepage twice before finding it.
Consider providing a more obvious logo in the header.
Consider increasing the size of the ‘Home’ link in the bread crumb trail and making it more obvious that it is clickable.
60 - Popular pages well used (Positive)
Most participants used the links in the ‘Popular pages’ section.
61 - Navigation behaviour (Observation)
Some participants said that they always returned to the homepage before starting a new task because it helped them to orientate themselves and find their way around websites.
“Yeah it’s my default back, it’s a natural. Get back to what I know and get back to the start.”
62 - External links not leading where expected (Major)
Users thought external links didn't always link to the page on the external website that had the answer. Participants still had to search on the external website to find the
“I'm confused, would have preferred for the exemption form to come up straight away.”
“I guess when you’re going around in circles which I found. That was the frustrating part about it. The links could be more accurate.”
“It takes me to another site like Work and Income or [Ministry of] Justice then it’s like you’ve jumped into another world.”
“Find that annoying - on one page and taken to another one, don't feel safe and accurate.”
Try to eliminate the need for users to go to another website to find the information they need.
Ensure that the link label describes precisely where the link will take users.
Try to make sure that links lead precisely to where users need to go.
63 - External links are fine (Observation)
Most participants didn't seem to have a problem with being redirected to another site.
“Doesn't bother me, as long as it takes me to where I want to be.”
“Doesn’t really bother me, as long as the information is there - doesn't bother me what site.”
“I don't mind being taken to another site. It took me to the actual people or website that can actually help me. “ 20-29 year old (Pacific Islander) from Auckland
64 - Unpredictable external links (Major)
Some participants were surprised when clicking a link, e.g. 'Apply to be excused from jury service' took them to another site.
Some participants didn't seem to realise when they were on another website.
“What's this website? Oh that's CAB - that's cool.”, 50-65 year old (Pacific Islander) from Auckland
“Oh, we’re back to small text again. Ah, it’s [agency name]. Miserable sods. They just want to save on ink.”
Consider using consistent and unique formatting for links that lead to other website, so user will know before clicking on a link that it is an external link
What we're doing
We'll be making some changes to the way the search engine indexes content on the site. We've also made sure the search engine is indexing the names and roles of the Ministers so there should be less no-results pages occurring in the future.
65 - Too many ‘No result’ searches (Severe)
Participants often use search terms as they would in Google, but multiple search terms, like 'Minister for IRD' or 'how to lodge a complaint' usually resulted in no-results pages.
“The search engine just doesn't like me.”
Ensure the site search caters for multiple search terms.
Ensure a 'No results found' message includes recommendations and suggestions how to improve the search.
What we're doing
The next update to the look and feel of the site includes an icon beside all external links to prompt users. We'll look at this feature again in later usability tests to see if this makes a difference.
66 - Poor handling of misspelling (Major)
The search didn’t handle misspellings very well; in general resulting in a ‘No results found’ message.
Some participants said they didn’t use search because they were not good at spelling.
For example, one participant didn’t complete a task, because she misspelled ‘warrenties’ when using the site search.
“I'd rather just go through the list; I might not be able to spell it right.”
Ensure that the site search can handle common spelling mistakes, offering correct spelling suggestions and subsequent search results.
67 - User search preferences (Observation)
The reasons why participants did or didn’t use the site search were manifold.
Most participants said they would only use search if they already knew what they were looking for; otherwise they would browse.
Some didn't use the site search because of bad experiences with other site searches in the past.
Especially younger participants used the search because they thought it was the quickest way.
“I was looking at this and it looked boring and not relevant to what I was wanting, didn’t want to read all the headings. Rather do it the fast easy way.”
“Ah, when I search for stuff it gets complicated and it throws lots of stuff at you. I usually just browse then if I need to I search.”
“I had a small idea already what I was looking for. If I had no clue, I wouldn’t search.”
68 - Descriptive search results (Positive)
Participants found the synopses under the search result headings useful for identifying the correct search result, e.g. that Ministry of Fisheries is now part of Ministry for Primary Industries.
69 - Missed information because didn’t scroll (Major)
Some participants missed information such as the ‘Government A-Z’ section because they didn’t scroll the page.
Some said they didn’t feel the need to scroll further down because all the things they needed were at the top of the page.
Place the key information at the top of the page such as the ‘Government A-Z’ section but make it obvious that there is more information further down.
Consider including elements that are longer than one screen length.
Remove horizontal lines and decrease amount of whitespace between the sections.
70 - Helpful headings and synopses (Positive)
Most participants found the headings and short synopses below the headings helpful and descriptive.
They thought it gave a good overview about what the website had to offer.
“It’s pretty easy to pick up what you are looking for.”
“Good it shows everything that it offers.”
“Everything on the homepage is easy to read, just a few clicks and you're where you want to be.”
“I like the headlines and underlining them - it's quite an accessible way of laying them out.”
71 - Confused by 'Green box' links (Minor)
Two participants selected the 'Departments & Agencies' link in the green box but got confused when it just took them further down the homepage.
“It didn’t take me where I thought, it though it would give me a list, I just didn’t see it.”
Consider making the ‘Departments & Agencies’ link a direct link to the ‘Government A-Z’ page.
72 - Didn’t see site search (Minor)
Some participants said that they didn’t see the search function
Consider keeping the search box always visible by not scrolling the header.
What we think
Given users preference for Google search over local search, this might not be of value to a large number of users. We won't implement this just yet, we need to look at how this would affect the overall responsive design we're using. eg, not scrolling the header on a mobile phone sized screen isn't a good idea it takes up too much space, so maybe this is a feature we could consider for larger tablet and desktop screen sizes.
Signpost Page - Level 4
73 - Missed information because didn’t scroll (Major)
Some participants missed information because they didn’t scroll or didn’t scroll all the way to the bottom of the page.
Show 'scroll-ability'. Consider including page elements that stretch over more than one page length.
Consider removing horizontal lines that reach across the whole page width.
Government A-Z page (aka Contact Government)
A priority area for us to look at
What we're doing
We've seen lots of users struggle with the A-Z page and the filter and the evidence from our Google Analytics data shows that people just don't understand the layout of the current page. We're going to give this page a complete overhaul and try something that's a bit different.
74 - Filter is ‘invisible’ (Major)
Most participants didn’t see the filter option because they were already concentrating on the list of department names.
Consider reducing the amount of text and making the label more succinct.
Consider using the same formatting for the filter label as for the list labels.
Consider making it visually more obvious that the filter function relates to the list.
75 - Filter by subject didn’t work (Major)
Most participants who used the filter used the department name, e.g. 'work and income' or a subject related to the department, e.g. 'fisheries', 'fishing', 'accommodation supplement'.
Participants didn't see the list getting shorter as they were typing because they concentrated on the keyboard or were typing too fast for the filter to respond, and got annoyed when they were left with a ‘0 matches’ message.
Consider enabling users to use subject-related terms to filter the 'Government A-Z' list.
76 - Thought filter is a search (Major)
Most participants thought the filter was a search.
Some said that’s why they didn't want to use it, based on bad experiences in the past, where site searches returned lots of irrelevant results.
“I would think it would search the same thing but it doesn’t. It's like a search within a search.”
“Yeah, it’s a search box, its searching within this page, but I didn’t need to cos it was right there.”
Shorten the filter label and include the term filter.
Consider emphasising that this feature will only 'search' within the page.
Consider replacing the filter text field with a different type of filter.
77 - Filter not saved (Minor)
One participant got confused when going back to the 'Government A-Z' page and the list wasn't filtered anymore.
Consider saving the user’s last filter settings.
78 - Scrolling quicker than filtering (Observation)
Most participants thought scrolling was quicker than filtering and most found the department/agency they were looking for, sometimes after scrolling up and down several times.
“[scrolling] is quicker than typing.”
“I knew it was Work and Income so would just scroll, it’s a one-hand operation, and not a two-hand one.”
79 - Confusing why right panel is not always there (Major)
Participants noticed and used the contact details in the right panel, but got confused if they weren't there.
Consider including contact details or a link to contacts on all pages.
80 - Phone number too generic (Minor)
One participant didn't think the phone number in the right panel is not the one she is looking for because it seems too generic.
“Yeah I did see those, but there’s a lot of departments within Ministry of Primary Industries and I thought it was just more general”
Consider renaming the ‘Contact page’ link to ‘More contact details’.
81 - Changing footer confusing (Major)
Participants didn’t understand why the footer content was different on different pages.
This particularly caused problems when participants wanted to use a footer link they had
seen before and couldn’t find it anymore, e.g. link to a specific government department.
“Ministry for Primary Industries is here, but wasn’t before. Why? I’m not sure why they would do that.”
“I don’t know. They're obviously not all there. I don’t why."" 50-65 year old (NZ European) from Dunedin
Consider providing the same sections, i.e. 'Information and Services, 'Contact Government', 'About this Site' and 'Site menu' on all pages.
Consider adding a 'More departments...' link to the 'Contact Government' section that will take users to the 'Contact Governments' page.
Consider renaming the 'Government A-Z' section in the footer to 'Departments related to this page content' or something similar.
What we're doing
In the next update to the look and feel of the site the footer will have a consistent set of links across all pages.
82 - Hyperlinks don’t look like links (Minor)
Some participants didn't think the 'Contact Government' heading in the footer was a link.
Some tried to click the 'Site menu' heading in the footer and got confused when it wasn't clickable.
Consider making all main headings in the footer clickable or use different text formatting to differentiate between clickable and static text.
83 - Contact details in footer (Minor)
Some participants looked for contact details in the footer and got confused when they couldn’t find it there, particularly on pages which didn’t have the contact details in the right panel.
“I didn’t see it first off in the side. Was looking for contact in this area [footer]. This part [right panel] is where you see the ads. I just don’t usually look there.”
“Quite often I go right to the bottom on the page for the contact us, but sometimes I go there and it’s a waste of time.”
Consider keeping the heading ‘Contact government’ in the footer.
84 - Footer is a shortcut (Positive)
Some participants preferred to use the footer because they saw it as a shortcut that allowed them to get quicker around the website.
“The panel at bottom turned out to be more helpful than I thought, it was driven by time.”
“Basically everything that up above on the home page, it’s just a shortcut basically.”
“This would make it easier if you know what you’re looking for; compared to the other that has the small paragraph.” Under 20 year old
(Asian) in Porirua
85 - Missed footer because didn’t scroll (Observation)
Some participants didn't see the footer because they didn’t scroll to the bottom of the page.
86 - Footer font too small (Minor)
Some participants commented that the font in the footer was too small.
Consider increasing the font size in the footer.
Topics page (aka Browse govt.nz)
87 - Hyperlink ‘hidden’ in footer (Major)
Only one participant saw the ‘Topics’ link in the footer. Most participants didn’t scroll this far down or overlooked the link.
Consider displaying 'Browse govt.nz' as a main heading in the footer, i.e. same style as 'About this site'.
88 - Good page layout (Positive)
Participants liked the layout of the ‘Topics’ page and preferred layout of browse.govt.nz to the homepage layout, because seeing everything on one page and not having to jump back and forth between pages was quicker.
“Prefer this one, it’s easier to read, and it’s listing everything.”
“I prefer this [one]. This is how I work. I don’t mind scrolling, but the other is more of a compact version, it just depends.”
“This is probably easier. Your eyes don’t have to go left/right left/right.”
Consider making browse.govt.nz layout an alternative homepage layout option.
89 - ‘Topics’ heading too generic (Minor)
Participants weren’t sure what kind of information would be hiding behind a heading like ‘Topics’. Most thought it would be FAQs.
“Topics is short for shortcut.”
- Last modified: