To try to keep this detailed write-up of our usability testing more manageable we've broken it down into 3 posts. This is the first one.
Introduction / Background
The initial implementation of Primo took place in mid-July 2010, and the usability testing took place within 2-3 weeks of this implementation after staff had attended a two-day Ex Libris training session. The aim of the testing was to see how intuitive the Primo interface was, and to gauge how useful students thought some of the additional functionality would be e.g. Facets, merging of records and FRBRisation, Reviews & Tags.
In terms of recruiting for the testing we knew that it would be difficult to get students in the middle of the summer vacation, however we used the details that we had taken during the name survey earlier in the year. This did give us all of our student testers, and we recruited 2 members of academic staff by asking Subject Librarians to contact lecturers they knew were helpful. We conducted 10 tests in total, with 7 of the testers coming from the Business, Environment & Society faculty, and 3 coming from Engineering & Computing. We were unable to recruit students from the Health & Life Sciences faculty, possibly due to course structures and placements. All testers had used the library catalogue and/or the eLibrary (MetaLib) before, but had mixed library search skills.
• 2 x academic staff
• 5 x undergraduate students (two 2nd years and three 3rd years)
• 3 x postgraduate students
We conducted the usability tests in a usability lab within the university. The lab allowed us to record audio and video of the participants, along with a close-up screen capture which showed how the participants navigated around the screen. Within the analysis below, only the audio and screen-captures were used.
To become familiar with the usability lab environment, and to test the questions that we had written, we asked a member of library staff with some experience of enquiry work to run through the questions for us as a pilot study. This provided valuable feedback on the wording of some questions which we were able to amend before beginning the testing for real.
The tests themselves consisted of 12 scripted questions along with time for discussion at the end (including questioning about the location of facets and the participants' understanding of Reviews & Tags). Each test took around 45 minutes to complete, including discussion. We tried to use scenario-based questions so that students would feel that the tasks they were being asked to do would represent how they would approach the library search interface in 'real life'.
Analysis and Recommendations
Section 1 - Searching for books and e-books
Question 1 - You want to find out if the Library has the book “Operations management” by Nigel Slack from your reading list. Can you check if the Library holds this book?
All 10 testers were able to complete the task of recognising whether the library stocked a particular book, however 4 testers required prompting to ensure that they had seen that we held the specific book in question. Of these 4 testers, 2 picked the same titled book by a different author, 1 picked a differently titled book by the same author, and 1 became distracted by the ‘view 3 versions’ link of a different book. 5 of the 6 testers who completed the task without prompting were able to do so in less than 1 minute. It was noted that all testers recognised the search box immediately, and 1 tester commented about being in the Books etc tab already.
Question 2 - Can you tell us what floor it is on and what number it’s at in the Library?
In this question we wanted to see where the testers would find the location of the book from on the screen. 9 out of 10 testers were able to give verbal confirmation of the floor and number that the book was held at. The other tester provided confirmation of the floor, but not the classmark. The majority of testers found this information in the availability line in either the brief or full screen. 2 testers found this information in the locations tab. 2 testers needed prompting to give either the floor or the classmark, and the average time taken to find this information was around 19 seconds.
Question 3 - Can you tell us how many copies are out on loan?
Only 3 of the testers managed to complete the task of finding out how many copies were on loan without prompting. 4 testers were unable to complete the task without prompting – many clicked on all tabs except the Locations tab. Some testers commented that they would not look there as it would only tell you where the book was held and not how many we had. Many testers thought that these details would be in the Details or More tabs. There were 3 testers who were unable to complete the task. One of these was due to not recognising the ‘expand’ buttons in the locations tab when there was more than one sublibrary. 3 testers thought that the ‘availability line’ status was clickable and that this was all the information there was available.
Recommendations:
• Re-label the locations tab to something that is more intuitive e.g. one tester suggested ‘availability’ – the ‘locations’ tab has been re-named as ‘Availability’.
• Ensure that if there are multiple sub-libraries that they expand automatically so that all copies are displayed – Ex Libris responded that this is not possible at this time.
• For Ex Libris: possibility of making the availability line status as clickable, taking users into the ‘locations’ tab?
Question 4 - If all copies of this edition were out on loan, can you see if there are any other editions available to borrow and if so how many editions?
9 out of 10 testers completed the task by acknowledging the ‘View x versions’ link and providing confirmation of the number of editions available. Of these, 4 testers had already clicked on the ‘View x versions’ link as part of the previous task and had already noted that it took them to a list of editions of the book. This indicated that we may need to change the labelling of the link to ensure that students would know that it meant ‘editions’ of the book and not ‘copies’. The one tester who did not complete the task was unsure of what an ‘edition’ meant and did not notice the link in either full or brief details screens. During test 7, the ‘View x versions’ link disappeared from the full details screen and so the tester was prompted to return to the brief results. Other testers had seen this link when the system was working correctly.
Recommendations:
• Re-name ‘View x versions’ as ‘View x editions’ – this has been done.
• Ensure that the edition statement appears within the record – we have added this to the ‘Details’ tab.
Question 5 - You are at home and need to find some books and don’t have time to come into the Library. Can you find a list of e-books on the subject of human resource management?
5 of the 10 testers were able to find a list of e-books using the e-book facet on the results screen, 4 testers were able to complete the task with prompting and one tester did not complete the task, but then found the answer whilst completing task 6. Several testers used the ‘Show Online Resources’ link but did not notice that this included journals and e-books. Several testers tried to find an ‘e-books’ option before attempting their search, 3 testers used the advanced search and looked in the format option expecting there to be an ‘e-book’ choice. 3 testers thought that e-books would be in the e-shelf area and were confused by this. 7 of the testers saw the e-book facet – 4 without prompting.
Recommendations:
• Include brief instructions to users on the front page telling them that they can refine their search after retrieving a list of results.
• Re-name ‘e-shelf’ as this caused confusion as testers believed this was where they would find e-resources – this has been re-named as My Favourites.
• Explore further ways of distinguishing between journals and books in the ‘Show Online Resources’ results list
Question 6 - Please try to access one of these items
In this task we wanted to see whether users would be able to see how to open an e-book from the results list, and whether they would be happy with viewing an e-book in the ‘letterbox’ preview screen that is automatically opened when a user clicks on ‘View Online’. There were two potential ways of opening the e-book into a bigger screen – an icon on the right-hand side that would enlarge the e-book into a bigger Primo window (the same view as would happen if a user was in the full-details screen rather than the brief results list view). The second option was a link on the left-hand side that said ‘Open Source in a New Window’ (OSINW) which would open the e-book in a full non-Primo window. Firstly, all of the testers noticed the ‘View Online’ link when either in the brief or full details screens and understood this to mean opening the e-book. Almost all of the testers were unhappy about attempting to view the e-book in the small letterbox view, and many were also reluctant to view the e-book in the bigger Primo window. 5 testers from the brief view clicked on the icon to open the e-book in a bigger screen. Only one tester saw the OSINW link from the brief details screen. Only 4 testers in total saw the OSINW link in either the brief or full details screens. 4 testers attempted to use the internal options within the e-book itself to try and make the e-book bigger – several testers became quite frustrated at not being able to open the e-book in a non-Primo window.
Recommendations:
• Configure Primo to ensure that when clicking on ‘View Online’ the e-book would automatically open in a full-screen, non-Primo window (preferred option) – this is being looked in to by Ex Libris and a fix is due to be released in an upcoming Service Pack.
• If above not possible – swap the links from the OSINW and icon options so that the icon would open a non-Primo window and the OSINW link opened in a Primo window (or preferably both open in a non-Primo window).
• Possibly attempt to re-name OSINW text to something clearer
Thursday, 21 October 2010
Tuesday, 21 September 2010
COVPRIMO one minute blog
At Coventry University we have been implementing Primo (which we are branding as 'Locate'), Ex Libris's next generation Library Catalogue.
The main driver behind launching a next generation OPAC was to provide a single user interface for staff and students to use as the starting point for all their Library searches, no matter what type of content they are looking for. This will help provide a more consistent user experience and improve the efficiency of information retrieval. This should help to increase the use of the Library's e-resources. The new system should also help us to provide an attractive interface for our users to access quality information and we believe it will be a viable alternative to systems such as Google, Wikipedia, etc.
The output from our JISC Project will be a report into our implementation of Primo, which should be useful for other institutions considering launching a next generation OPAC and particularly for those deciding to implement Primo themselves. The report will also include an overview of the usability testing we carried out as part of our implementation. This testing was carried out with staff and students to try to identify ways in which we can make our Primo more intuitive and user-friendly. We have then made changes to the system based on these tests. We also gained further student input via a focus group where we discussed what is the key information to include on the Primo home page and how should this be laid out.
The general feedback we received about Primo was overwhelmingly positive, but we did identify issues which needed changes to be made, as they caused problems for several of the users involved in the testing. We have made numerous changes to the system already and we're waiting on Ex Libris to resolve some support issues before making other changes. We are also feeding back the resuts of our tests to Ex Libris and have been told that they will be taking these on board. We hope this means that where we discovered issues that cannot currently be changed Ex Libris will consider adding functionality that we require to future releases of Primo.
Implementation notes
Two IBM rack mounted servers [x3550 M2 8 core processing 16Gb RAM running Red Hat Linux 5.5 64 bit] were purchased and commissioned, one acting as a staging server, the other as the production server. In our case the supplier's software installation was carried out on the production server initially with a copy made to staging fairly late on in the project - normally this procedure is reversed. Project management was handled through the supplier's German office with conference calls established as and when required; these calls tended to increase in length and frequency the closer we moved towards our target switch-to-production (STP) date. On-site staff training was carried out from Ex Libris' UK office and was generally well received by staff.
Once the application had been installed, there were four main thrusts to the implementation work: the design of the end-user interface, taking on board the feedback received as part of the usability study; the testing of data export / import routines from our existing systems (Aleph, MetaLib, SFX and the University's repository product Equella); the establishing of automated data update routines and finally the installation / configuration of third party ancillary software products.
We were aware at the start of the project we had set ourselves an ambitious 'go live' date of early September, hoping to allow subject staff some time to prepare for the forthcoming student induction. In the event a decision has been taken to postpone the launch on two counts. Firstly Shibboleth configuration work is still incomplete and secondly Primo Central functionality is now scheduled to come on stream towards the end of October; Locate will be launched in the new year.
Our JISC project co-ordinator James Fisher leaves Coventry University on Friday 24 September 2010; we wish James well in his new post at the Univeristy of Warwick. Graham Daniels takes over as project contact following James' departure.
The main driver behind launching a next generation OPAC was to provide a single user interface for staff and students to use as the starting point for all their Library searches, no matter what type of content they are looking for. This will help provide a more consistent user experience and improve the efficiency of information retrieval. This should help to increase the use of the Library's e-resources. The new system should also help us to provide an attractive interface for our users to access quality information and we believe it will be a viable alternative to systems such as Google, Wikipedia, etc.
The output from our JISC Project will be a report into our implementation of Primo, which should be useful for other institutions considering launching a next generation OPAC and particularly for those deciding to implement Primo themselves. The report will also include an overview of the usability testing we carried out as part of our implementation. This testing was carried out with staff and students to try to identify ways in which we can make our Primo more intuitive and user-friendly. We have then made changes to the system based on these tests. We also gained further student input via a focus group where we discussed what is the key information to include on the Primo home page and how should this be laid out.
The general feedback we received about Primo was overwhelmingly positive, but we did identify issues which needed changes to be made, as they caused problems for several of the users involved in the testing. We have made numerous changes to the system already and we're waiting on Ex Libris to resolve some support issues before making other changes. We are also feeding back the resuts of our tests to Ex Libris and have been told that they will be taking these on board. We hope this means that where we discovered issues that cannot currently be changed Ex Libris will consider adding functionality that we require to future releases of Primo.
Implementation notes
Two IBM rack mounted servers [x3550 M2 8 core processing 16Gb RAM running Red Hat Linux 5.5 64 bit] were purchased and commissioned, one acting as a staging server, the other as the production server. In our case the supplier's software installation was carried out on the production server initially with a copy made to staging fairly late on in the project - normally this procedure is reversed. Project management was handled through the supplier's German office with conference calls established as and when required; these calls tended to increase in length and frequency the closer we moved towards our target switch-to-production (STP) date. On-site staff training was carried out from Ex Libris' UK office and was generally well received by staff.
Once the application had been installed, there were four main thrusts to the implementation work: the design of the end-user interface, taking on board the feedback received as part of the usability study; the testing of data export / import routines from our existing systems (Aleph, MetaLib, SFX and the University's repository product Equella); the establishing of automated data update routines and finally the installation / configuration of third party ancillary software products.
We were aware at the start of the project we had set ourselves an ambitious 'go live' date of early September, hoping to allow subject staff some time to prepare for the forthcoming student induction. In the event a decision has been taken to postpone the launch on two counts. Firstly Shibboleth configuration work is still incomplete and secondly Primo Central functionality is now scheduled to come on stream towards the end of October; Locate will be launched in the new year.
Our JISC project co-ordinator James Fisher leaves Coventry University on Friday 24 September 2010; we wish James well in his new post at the Univeristy of Warwick. Graham Daniels takes over as project contact following James' departure.
Monday, 6 September 2010
Focus Group - Primo Home Page
In order to get student feedback about what information should be on our Primo home page and the layout of the home page we arranged a focus group with ten students at the end of August. To get our participants we first emailed all of our volunteers who had given us their email addresses back in June. We had a much lower response rate than when we were arranging the usability tests. So to get more participants we went round the Library asking students if they would be available to come along to our group. We soon reached our target of ten students, and were very pleased that they all turned up at the appointed time.
The session was extremely productive with all the students contributing and lots of ideas being discussed. To start the session the three members of Library staff introduced themselves and then the ten students did the same and told the group what they were studying. To get everyone warmed up we asked them to tell us what they do first of all when they get an assignment and have to find relevant information. This produced a lively discussion and despite a number of students using Wikipedia and Google it was reassuring to find that everyone made use of the Library’s subscribed resources too and most seemed to use MetaLib to get to the databases. There was a quick debate around the pros and cons of Wikipedia. Most students agreed that they shouldn’t reference it and many had been given strict instructions about this by their lecturers. However, a number of students found it a useful starting point and used it to find search terms about their topic to enter into the Library databases. This was an interesting aside and seemed to get everyone involved from the very start of the session.
We then gave a quick overview of what to expect from the rest of the session. We also showed the group what the Coventry Primo currently looks like and did a quick demo of a search. We had some immediate positive feedback at this point about the look of the Primo interface. It was described by one user as “very easy on the eye.” Then we told them about the usability studies that had already taken place, and highlighted one of the changes that we had made because of these studies.
We then asked the participants to suggest as many ideas as they could think of that they would like to see on the home page. The list of suggestions that the group came up with and agreed would be useful on the home page was as follows:
· Announcements (e.g. work going on in the Library at the moment)
· Room booking / availability
· Search hints, tips / Search examples
· Introduction to Primo – Tutorial
· Contacts, Profiles, Who’s Who, Who’s responsible for certain subject areas
· Advertising for the University as a whole – Student Centre, Social Events, etc.
· Opening times
· Advanced Search
· Contact IT Services for help with your password, or with signing in
· Feedback section for Primo or for anything to do with the Library
· Ask a Librarian
· Latest books added to Library stock in different subject areas
· Passwords help, change password information
· Image of the Library (maybe as a watermark with links on top)
· A video with search hints
· Autocomplete functionality when typing searches
· Link to the Centre for Academic Writing & referencing information
· Library FAQs
· CU Portal link
· CUOnline / Moodle link
We then wrote each of the suggestions on post-its and worked on a flip-chart with a hand drawn version of the Primo home page. This had five areas on it. These were the header area which includes the search box, and the footer area. These two areas will remain fixed as users carry out searches. The rest of the home page is divided into three parts: a top left and top right section and a bottom section which goes across the whole page. The group suggested whereabouts on the home page was the best place to put each of their suggestions. There was quite a bit of debate around where was the best place to put each suggestion and some ideas were even rejected and removed from the list. In the end a consensus was reached with similar items being grouped together.
It was suggested that Announcements could appear in a ‘ticker’ format scrolling across the screen. This was something to look at in the future. There was a big debate around where on the page they should appear. It was agreed this should be limited to just Library announcements, and nothing else, and maybe just the top few announcements and headlines only to keep it brief. Eventually it was agreed that links to announcements could go on the top right of the page, along with new books added to stock and room bookings information.
Search hints should be at the top on the left under the search box. Also a short introduction to Primo should be here and possibly also a link to a video introduction, which is something else to look at in the future. An Ask a Librarian link should be in this area too. There was agreement to add a link to the text at the end of the search hints.
The Ask a Librarian link should also be in the header, so it is always available. The header and footer stay on the screen when a search is carried out, so would always be available, whereas the rest of the home page would be replaced by search results, etc. There was also consensus that a link to Library Passwords information should be in this section.
Contact information for Subject Librarians could go in the footer as this is a common place to find contacts. Links to IT Services & general ITS Passwords information could also go in this section, along with Library FAQs.
There was discussion over where to put the link to the student portal (CU Portal). Suggestions were to put this in the footer, but also to put a link to this at the top of the screen. Eventually we agreed to make the Coventry University logo at the top of the screen a link to CU Portal as this is the most likely place you would want to link to from Primo and also put another link in the footer.
It was suggested that the opening hours could go in the bottom section of the home page along with links to the Centre for Academic Writing and referencing.
The group thought it would be good if an image of the Library could be included on the page, possibly as a watermark. However, it was agreed that this should only be added if it didn’t distract users from the text on the home page.
We then looked at three other Primo Version 3 home pages, that are already live, and the group gave us input into what they liked and didn’t like about these sites.
After looking at the other sites it was agreed that the amount of text on the Coventry Primo home page should be kept to a minimum. One comment was “The simpler the better.” Ideally, the group wanted headings in bold that you could click on to make the information beneath those headers display. This would mean that initially there would just be headings on the page, which would prevent it from being too cluttered. However, it needs to be clear that this is what will happen if you click on the heading. The information also needs to be spread out enough to fill the screen.
The group also commented that the out-of-the-box image of books on a shelf, that displays on several Primo sites really drew the eye. However, this was probably the least important thing on the page. They still felt it could be good to include a watermarked image of our Library. However, we would need to be careful this didn’t detract from the information on the page if we included this.
Revisiting the mock-up of the home page it was again agreed to keep similar items grouped together but where possible to just display a general heading which you can click on in order to display the links beneath that heading. A picture of the final mock-up can be seen below:
One other point of interest that was raised several times during the session was the expectation of personalised information for individual users. Participants wanted information about previous searches, books that they have looked for or used before and new books in the Library relevant to their subject. We explained that much of this information is available in the area that we have named ‘My Favourites’. We didn’t go into too much more detail about this as we were focussing on the home page for all users, initially accessing it as a guest. However, it was clear that personalisation was something that should be looked at further in the future.
Overall it was an incredibly useful session. We will now sit down with the Library Systems Manager who is working on these kinds of developments in Primo and try to turn the comments into reality.
The session was extremely productive with all the students contributing and lots of ideas being discussed. To start the session the three members of Library staff introduced themselves and then the ten students did the same and told the group what they were studying. To get everyone warmed up we asked them to tell us what they do first of all when they get an assignment and have to find relevant information. This produced a lively discussion and despite a number of students using Wikipedia and Google it was reassuring to find that everyone made use of the Library’s subscribed resources too and most seemed to use MetaLib to get to the databases. There was a quick debate around the pros and cons of Wikipedia. Most students agreed that they shouldn’t reference it and many had been given strict instructions about this by their lecturers. However, a number of students found it a useful starting point and used it to find search terms about their topic to enter into the Library databases. This was an interesting aside and seemed to get everyone involved from the very start of the session.
We then gave a quick overview of what to expect from the rest of the session. We also showed the group what the Coventry Primo currently looks like and did a quick demo of a search. We had some immediate positive feedback at this point about the look of the Primo interface. It was described by one user as “very easy on the eye.” Then we told them about the usability studies that had already taken place, and highlighted one of the changes that we had made because of these studies.
We then asked the participants to suggest as many ideas as they could think of that they would like to see on the home page. The list of suggestions that the group came up with and agreed would be useful on the home page was as follows:
· Announcements (e.g. work going on in the Library at the moment)
· Room booking / availability
· Search hints, tips / Search examples
· Introduction to Primo – Tutorial
· Contacts, Profiles, Who’s Who, Who’s responsible for certain subject areas
· Advertising for the University as a whole – Student Centre, Social Events, etc.
· Opening times
· Advanced Search
· Contact IT Services for help with your password, or with signing in
· Feedback section for Primo or for anything to do with the Library
· Ask a Librarian
· Latest books added to Library stock in different subject areas
· Passwords help, change password information
· Image of the Library (maybe as a watermark with links on top)
· A video with search hints
· Autocomplete functionality when typing searches
· Link to the Centre for Academic Writing & referencing information
· Library FAQs
· CU Portal link
· CUOnline / Moodle link
We then wrote each of the suggestions on post-its and worked on a flip-chart with a hand drawn version of the Primo home page. This had five areas on it. These were the header area which includes the search box, and the footer area. These two areas will remain fixed as users carry out searches. The rest of the home page is divided into three parts: a top left and top right section and a bottom section which goes across the whole page. The group suggested whereabouts on the home page was the best place to put each of their suggestions. There was quite a bit of debate around where was the best place to put each suggestion and some ideas were even rejected and removed from the list. In the end a consensus was reached with similar items being grouped together.
It was suggested that Announcements could appear in a ‘ticker’ format scrolling across the screen. This was something to look at in the future. There was a big debate around where on the page they should appear. It was agreed this should be limited to just Library announcements, and nothing else, and maybe just the top few announcements and headlines only to keep it brief. Eventually it was agreed that links to announcements could go on the top right of the page, along with new books added to stock and room bookings information.
Search hints should be at the top on the left under the search box. Also a short introduction to Primo should be here and possibly also a link to a video introduction, which is something else to look at in the future. An Ask a Librarian link should be in this area too. There was agreement to add a link to the text at the end of the search hints.
The Ask a Librarian link should also be in the header, so it is always available. The header and footer stay on the screen when a search is carried out, so would always be available, whereas the rest of the home page would be replaced by search results, etc. There was also consensus that a link to Library Passwords information should be in this section.
Contact information for Subject Librarians could go in the footer as this is a common place to find contacts. Links to IT Services & general ITS Passwords information could also go in this section, along with Library FAQs.
There was discussion over where to put the link to the student portal (CU Portal). Suggestions were to put this in the footer, but also to put a link to this at the top of the screen. Eventually we agreed to make the Coventry University logo at the top of the screen a link to CU Portal as this is the most likely place you would want to link to from Primo and also put another link in the footer.
It was suggested that the opening hours could go in the bottom section of the home page along with links to the Centre for Academic Writing and referencing.
The group thought it would be good if an image of the Library could be included on the page, possibly as a watermark. However, it was agreed that this should only be added if it didn’t distract users from the text on the home page.
We then looked at three other Primo Version 3 home pages, that are already live, and the group gave us input into what they liked and didn’t like about these sites.
After looking at the other sites it was agreed that the amount of text on the Coventry Primo home page should be kept to a minimum. One comment was “The simpler the better.” Ideally, the group wanted headings in bold that you could click on to make the information beneath those headers display. This would mean that initially there would just be headings on the page, which would prevent it from being too cluttered. However, it needs to be clear that this is what will happen if you click on the heading. The information also needs to be spread out enough to fill the screen.
The group also commented that the out-of-the-box image of books on a shelf, that displays on several Primo sites really drew the eye. However, this was probably the least important thing on the page. They still felt it could be good to include a watermarked image of our Library. However, we would need to be careful this didn’t detract from the information on the page if we included this.
Revisiting the mock-up of the home page it was again agreed to keep similar items grouped together but where possible to just display a general heading which you can click on in order to display the links beneath that heading. A picture of the final mock-up can be seen below:
One other point of interest that was raised several times during the session was the expectation of personalised information for individual users. Participants wanted information about previous searches, books that they have looked for or used before and new books in the Library relevant to their subject. We explained that much of this information is available in the area that we have named ‘My Favourites’. We didn’t go into too much more detail about this as we were focussing on the home page for all users, initially accessing it as a guest. However, it was clear that personalisation was something that should be looked at further in the future.
Overall it was an incredibly useful session. We will now sit down with the Library Systems Manager who is working on these kinds of developments in Primo and try to turn the comments into reality.
Friday, 3 September 2010
Usability Testing Primo
About a month ago we carried out our usability tests on the out-of-the-box version of Primo. This had been set up to use data from our Aleph Library Management System test server. We also had data from MetaLib, our Federated Search Tool and SFX our Open URL Resolver harvested into Primo. We wanted to get input from staff and students on things that they found unintuitive, so that we could try to make improvements to the system before launching it.
Before starting testing we visited the University’s usability lab and talked things through with the technician. We also enlisted the help of an experienced usability tester, a PhD student from the University who had spent a lot of time running tests as part of her research. She gave us plenty of helpful advice and even attended our pilot study to give us feedback on ways to improve our test before running it with students and academic staff.
We ran the pilot study using a member of Library staff who had not seen our Primo installation at all, so had no preconceptions. Carrying out this pilot enabled us to familiarise ourselves with the environment in the usability lab and also to check that our questions all seemed to work and make sense to the tester. We did identify some changes to make to the questions before carrying out the testing for real.
We recruited 8 students and 2 members of academic staff to carry out the usability tests. We had obtained about 40 students’ contact details earlier in the project when carrying out a survey on what to call our Primo. This meant we were able to send emails to the volunteers with the incentive of paying for their time if they came in and worked with us. Although it was an unfortunate time to carry out the tests right in the middle of the summer vacation we still managed to recruit our 8 students without too many problems. Unsurprisingly we didn’t receive responses from a number of emails, but we were able to continue sending out requests until we had enough volunteers. Pleasingly, although we had to rearrange a couple of the tests at the request of the students everyone who volunteered did attend and arrived on time. The members of staff that we recruited were regular Library users who were suggested by the Subject Librarians who knew them.
We allowed an hour to carry out each test and to have a discussion afterwards and found that a number of different themes emerged, which helped us identify changes we wanted to make in order to make the system more user-friendly. Firstly there were things to do with the appearance of the interface, which we felt could be changed to make it more intuitive. For example, after carrying out a search the facets available to refine that search are displayed in a column on the left-hand-side of the screen. However, there is not much differentiation between these facets and the search results that display in the centre of the screen. Everything is shown on the same white background and there is only a faint line between the facets and the results. This meant that a number of our testers didn’t spot the facets for quite some time. Once they did spot them they did understand what they were for and used them intuitively to narrow down their search. We had a number of pleasing comments about how obvious it was what the facets did and also comparisons with Tesco Online and Amazon in the way that they worked. This showed how important it is to make the facets prominent to the users as they did find them useful once they noticed them. Therefore, in order to try to make them stand out more we added shading to the facets to differentiate them from the rest of the page and made the headings bolder. We also made the line between the facets and the results more defined. We believe this will make the facets easier to spot.
The second theme that emerged was to do with terminology that was used. An example of this is that the personalised area of Primo where users can save search results or previous searches, etc. was originally called the e-Shelf. We had a number of testers who thought that this would be where they would find e-books, which does have a certain logic to it: An e-book would be on an e-Shelf. Several other testers were also confused by the terminology and kept returning to this area, even though it wasn’t relevant to any of the tasks they have been set. Therefore, in order to try to avoid this confusion we have renamed the e-Shelf as ‘My Favourites’ as we believe this is clearer.
Another theme that emerged was to do with the functionality of Primo. For example when Primo finds multiple editions of an item it groups them all together under the Functional Requirements for Bibliographic Records (FRBR) rules. However, when searching by relevance it always prioritises the online version, even if this isn’t the most recent. In our experience with e-books, publishers often only make an electronic version of a title available some time after they have published the print version. This delay can be 6 months or it could be a year or more. This means that we often have new print editions of essential books available in the Library well before we are able to obtain an electronic version. We will often have previous print and electronic editions. This means that when Primo groups these records into a FRBR group the edition it chooses to display in the list of search results is the old electronic edition. The user won’t find out that we hold the most recent edition until they click on the FRBR group and examine the list of other available editions. When we asked Ex Libris if we could continue to search by relevance but make sure that the most recent edition of a book displays in the results list we were told this wasn’t standard functionality. Therefore, we had to raise a call to get our request investigated further and it’s possible that we won’t be able to make this change until a future version of Primo is released. We are waiting in hope for Ex Libris to investigate our request as we would like to make this change before we go live with Primo.
Most of the changes that we identified through the usability tests could be put into one or more of these categories. Since the tests both Paul here in the Library and Lee from Ex Libris have been working very hard to configure our Primo based on the results. It has already changed a great deal, but there is still more work to do. We have a consultancy day with Ex Libris here at Coventry next week, so we are hoping that we will be able to clear up many of the outstanding issues then.
Before starting testing we visited the University’s usability lab and talked things through with the technician. We also enlisted the help of an experienced usability tester, a PhD student from the University who had spent a lot of time running tests as part of her research. She gave us plenty of helpful advice and even attended our pilot study to give us feedback on ways to improve our test before running it with students and academic staff.
We ran the pilot study using a member of Library staff who had not seen our Primo installation at all, so had no preconceptions. Carrying out this pilot enabled us to familiarise ourselves with the environment in the usability lab and also to check that our questions all seemed to work and make sense to the tester. We did identify some changes to make to the questions before carrying out the testing for real.
We recruited 8 students and 2 members of academic staff to carry out the usability tests. We had obtained about 40 students’ contact details earlier in the project when carrying out a survey on what to call our Primo. This meant we were able to send emails to the volunteers with the incentive of paying for their time if they came in and worked with us. Although it was an unfortunate time to carry out the tests right in the middle of the summer vacation we still managed to recruit our 8 students without too many problems. Unsurprisingly we didn’t receive responses from a number of emails, but we were able to continue sending out requests until we had enough volunteers. Pleasingly, although we had to rearrange a couple of the tests at the request of the students everyone who volunteered did attend and arrived on time. The members of staff that we recruited were regular Library users who were suggested by the Subject Librarians who knew them.
We allowed an hour to carry out each test and to have a discussion afterwards and found that a number of different themes emerged, which helped us identify changes we wanted to make in order to make the system more user-friendly. Firstly there were things to do with the appearance of the interface, which we felt could be changed to make it more intuitive. For example, after carrying out a search the facets available to refine that search are displayed in a column on the left-hand-side of the screen. However, there is not much differentiation between these facets and the search results that display in the centre of the screen. Everything is shown on the same white background and there is only a faint line between the facets and the results. This meant that a number of our testers didn’t spot the facets for quite some time. Once they did spot them they did understand what they were for and used them intuitively to narrow down their search. We had a number of pleasing comments about how obvious it was what the facets did and also comparisons with Tesco Online and Amazon in the way that they worked. This showed how important it is to make the facets prominent to the users as they did find them useful once they noticed them. Therefore, in order to try to make them stand out more we added shading to the facets to differentiate them from the rest of the page and made the headings bolder. We also made the line between the facets and the results more defined. We believe this will make the facets easier to spot.
The second theme that emerged was to do with terminology that was used. An example of this is that the personalised area of Primo where users can save search results or previous searches, etc. was originally called the e-Shelf. We had a number of testers who thought that this would be where they would find e-books, which does have a certain logic to it: An e-book would be on an e-Shelf. Several other testers were also confused by the terminology and kept returning to this area, even though it wasn’t relevant to any of the tasks they have been set. Therefore, in order to try to avoid this confusion we have renamed the e-Shelf as ‘My Favourites’ as we believe this is clearer.
Another theme that emerged was to do with the functionality of Primo. For example when Primo finds multiple editions of an item it groups them all together under the Functional Requirements for Bibliographic Records (FRBR) rules. However, when searching by relevance it always prioritises the online version, even if this isn’t the most recent. In our experience with e-books, publishers often only make an electronic version of a title available some time after they have published the print version. This delay can be 6 months or it could be a year or more. This means that we often have new print editions of essential books available in the Library well before we are able to obtain an electronic version. We will often have previous print and electronic editions. This means that when Primo groups these records into a FRBR group the edition it chooses to display in the list of search results is the old electronic edition. The user won’t find out that we hold the most recent edition until they click on the FRBR group and examine the list of other available editions. When we asked Ex Libris if we could continue to search by relevance but make sure that the most recent edition of a book displays in the results list we were told this wasn’t standard functionality. Therefore, we had to raise a call to get our request investigated further and it’s possible that we won’t be able to make this change until a future version of Primo is released. We are waiting in hope for Ex Libris to investigate our request as we would like to make this change before we go live with Primo.
Most of the changes that we identified through the usability tests could be put into one or more of these categories. Since the tests both Paul here in the Library and Lee from Ex Libris have been working very hard to configure our Primo based on the results. It has already changed a great deal, but there is still more work to do. We have a consultancy day with Ex Libris here at Coventry next week, so we are hoping that we will be able to clear up many of the outstanding issues then.
Friday, 16 July 2010
Naming our Primo installation
One of the preliminary discussions that we had early on in our project was what to call our Primo installation. It soon became clear that there were two distinct options. We could either try to retain some continuity or we could choose a completely new name to indicate that this is a brand new system.
As we are planning to use Primo to completely replace our Library Catalogue and eLibrary (MetaLib) interfaces we could try to retain some continuity by using a slightly amended form of these names. To this end Library Catalogue Plus and eLibrary Plus were suggested. Certainly we have seen examples of other Primo sites where the word Plus has been added to the current Library Catalogue name, so this felt like a valid possibility.
The alternative option was to think of a brand new name and after much discussion we decided on two possibilities. These were SearchLight and LOCATe. We thought SearchLight could indicate that you were shining a light onto your search in order to find information. The name LOCATe could stand for Library Online Catalogue, and of course the word locate also means find. It was agreed that if either of these names were selected we would also need a strap line which would summarise what the system does.
As we were having these discussions just before the end of term we decided to try to get some student input into which name our users preferred. In order to try to reach as many people as possible we used two methods to get feedback. Firstly, we went around the Library and asked users if they could spare a few minutes to give us their views. To those that were happy to do this we explained briefly about the plan to introduce Primo and asked them which name they liked best. Secondly, we created a SurveyMonkey poll where we again explained about Primo and asked which of the four names respondents preferred.
We received just under 150 responses, split very evenly between online and face-to-face. Unfortunately, the results weren’t completely conclusive, although there were two names that came out ahead. These were eLibrary Plus and LOCATe and both had a similar number of votes. As term was ending and there wasn’t time to do another student poll it was decided to survey the Library staff and ask them to choose between these two options. This time there was a more conclusive result and the name LOCATe was chosen.
A very useful side benefit of carrying out the user survey about the name was that we also asked for volunteers to take part in our usability testing over the summer. We knew this was likely to take place in July or August, so we asked everyone who gave us input about the name if they lived close enough to the University to come in over summer and if so whether they were interested in taking part in some testing if there was a financial incentive. As a result of this we got 40 volunteers, all of whom gave us their email addresses and said we could contact them when we were arranging the testing. This was a big relief as one of our concerns had been how we would contact users about the tests during the quiet summer period.
As we are planning to use Primo to completely replace our Library Catalogue and eLibrary (MetaLib) interfaces we could try to retain some continuity by using a slightly amended form of these names. To this end Library Catalogue Plus and eLibrary Plus were suggested. Certainly we have seen examples of other Primo sites where the word Plus has been added to the current Library Catalogue name, so this felt like a valid possibility.
The alternative option was to think of a brand new name and after much discussion we decided on two possibilities. These were SearchLight and LOCATe. We thought SearchLight could indicate that you were shining a light onto your search in order to find information. The name LOCATe could stand for Library Online Catalogue, and of course the word locate also means find. It was agreed that if either of these names were selected we would also need a strap line which would summarise what the system does.
As we were having these discussions just before the end of term we decided to try to get some student input into which name our users preferred. In order to try to reach as many people as possible we used two methods to get feedback. Firstly, we went around the Library and asked users if they could spare a few minutes to give us their views. To those that were happy to do this we explained briefly about the plan to introduce Primo and asked them which name they liked best. Secondly, we created a SurveyMonkey poll where we again explained about Primo and asked which of the four names respondents preferred.
We received just under 150 responses, split very evenly between online and face-to-face. Unfortunately, the results weren’t completely conclusive, although there were two names that came out ahead. These were eLibrary Plus and LOCATe and both had a similar number of votes. As term was ending and there wasn’t time to do another student poll it was decided to survey the Library staff and ask them to choose between these two options. This time there was a more conclusive result and the name LOCATe was chosen.
A very useful side benefit of carrying out the user survey about the name was that we also asked for volunteers to take part in our usability testing over the summer. We knew this was likely to take place in July or August, so we asked everyone who gave us input about the name if they lived close enough to the University to come in over summer and if so whether they were interested in taking part in some testing if there was a financial incentive. As a result of this we got 40 volunteers, all of whom gave us their email addresses and said we could contact them when we were arranging the testing. This was a big relief as one of our concerns had been how we would contact users about the tests during the quiet summer period.
Tuesday, 15 June 2010
Budget
JISC require that an outline budget is posted, so here is the budget for the covprimo project:
Projected Timeline, Workplan & Methodology
The following chart details the main work packages as identified by Ex Libris in their Primo Implementation project plan and the timescales they are working to. Local tasks such as the usability testing and the launch and publicity have then been planned to take place at appropriate times.
Subscribe to:
Posts (Atom)
