Usability Testing Primo

About a month ago we carried out our usability tests on the out-of-the-box version of Primo. This had been set up to use data from our Aleph Library Management System test server. We also had data from MetaLib, our Federated Search Tool and SFX our Open URL Resolver harvested into Primo. We wanted to get input from staff and students on things that they found unintuitive, so that we could try to make improvements to the system before launching it.

Before starting testing we visited the University’s usability lab and talked things through with the technician. We also enlisted the help of an experienced usability tester, a PhD student from the University who had spent a lot of time running tests as part of her research. She gave us plenty of helpful advice and even attended our pilot study to give us feedback on ways to improve our test before running it with students and academic staff.

We ran the pilot study using a member of Library staff who had not seen our Primo installation at all, so had no preconceptions. Carrying out this pilot enabled us to familiarise ourselves with the environment in the usability lab and also to check that our questions all seemed to work and make sense to the tester. We did identify some changes to make to the questions before carrying out the testing for real.

We recruited 8 students and 2 members of academic staff to carry out the usability tests. We had obtained about 40 students’ contact details earlier in the project when carrying out a survey on what to call our Primo. This meant we were able to send emails to the volunteers with the incentive of paying for their time if they came in and worked with us. Although it was an unfortunate time to carry out the tests right in the middle of the summer vacation we still managed to recruit our 8 students without too many problems. Unsurprisingly we didn’t receive responses from a number of emails, but we were able to continue sending out requests until we had enough volunteers. Pleasingly, although we had to rearrange a couple of the tests at the request of the students everyone who volunteered did attend and arrived on time. The members of staff that we recruited were regular Library users who were suggested by the Subject Librarians who knew them.

We allowed an hour to carry out each test and to have a discussion afterwards and found that a number of different themes emerged, which helped us identify changes we wanted to make in order to make the system more user-friendly. Firstly there were things to do with the appearance of the interface, which we felt could be changed to make it more intuitive. For example, after carrying out a search the facets available to refine that search are displayed in a column on the left-hand-side of the screen. However, there is not much differentiation between these facets and the search results that display in the centre of the screen. Everything is shown on the same white background and there is only a faint line between the facets and the results. This meant that a number of our testers didn’t spot the facets for quite some time. Once they did spot them they did understand what they were for and used them intuitively to narrow down their search. We had a number of pleasing comments about how obvious it was what the facets did and also comparisons with Tesco Online and Amazon in the way that they worked. This showed how important it is to make the facets prominent to the users as they did find them useful once they noticed them. Therefore, in order to try to make them stand out more we added shading to the facets to differentiate them from the rest of the page and made the headings bolder. We also made the line between the facets and the results more defined. We believe this will make the facets easier to spot.

The second theme that emerged was to do with terminology that was used. An example of this is that the personalised area of Primo where users can save search results or previous searches, etc. was originally called the e-Shelf. We had a number of testers who thought that this would be where they would find e-books, which does have a certain logic to it: An e-book would be on an e-Shelf. Several other testers were also confused by the terminology and kept returning to this area, even though it wasn’t relevant to any of the tasks they have been set. Therefore, in order to try to avoid this confusion we have renamed the e-Shelf as ‘My Favourites’ as we believe this is clearer.

Another theme that emerged was to do with the functionality of Primo. For example when Primo finds multiple editions of an item it groups them all together under the Functional Requirements for Bibliographic Records (FRBR) rules. However, when searching by relevance it always prioritises the online version, even if this isn’t the most recent. In our experience with e-books, publishers often only make an electronic version of a title available some time after they have published the print version. This delay can be 6 months or it could be a year or more. This means that we often have new print editions of essential books available in the Library well before we are able to obtain an electronic version. We will often have previous print and electronic editions. This means that when Primo groups these records into a FRBR group the edition it chooses to display in the list of search results is the old electronic edition. The user won’t find out that we hold the most recent edition until they click on the FRBR group and examine the list of other available editions. When we asked Ex Libris if we could continue to search by relevance but make sure that the most recent edition of a book displays in the results list we were told this wasn’t standard functionality. Therefore, we had to raise a call to get our request investigated further and it’s possible that we won’t be able to make this change until a future version of Primo is released. We are waiting in hope for Ex Libris to investigate our request as we would like to make this change before we go live with Primo.

Most of the changes that we identified through the usability tests could be put into one or more of these categories. Since the tests both Paul here in the Library and Lee from Ex Libris have been working very hard to configure our Primo based on the results. It has already changed a great deal, but there is still more work to do. We have a consultancy day with Ex Libris here at Coventry next week, so we are hoping that we will be able to clear up many of the outstanding issues then.