An Unobtrusive Evaluation of Online Real Time Library Reference Services
by
Neal Kaske and Julie Arnold
for the
Library Research Round Table, American Library Association,
Annual Conference, Atlanta, Georgia,
June 15, 2002
This
empirical study’s main goal was to answer the following three questions. 1) How well is the new ‘Online Live
Reference’ or ‘Chat’ library service performing? 2) Does the 55% rule reported for past research into the level of
correct answers provided by reference services also hold true for this new
service? 3) What can we learn about the reference interview process through
studying the transaction logs for these services? In
order to meet this research goal, a set of 12 questions was developed and posed
unobtrusively to a sample of 36 ‘Chat’ services in late Fall of 2001. The same
questions were also posed to these 36 libraries’ central email services where
available. A total of 180 chat sessions
were attempted and 180 email questions were sent. The chat sessions successfully
completed numbered 133 and the percentage with correct answers was 55%. Email
questions answered totaled 107 and the percentage answered correctly was 60%.
The difference between questions ranged from 0.0% to 84.6% correct for chat
sessions and 14.3% to 100.0% for email.
Many insights into the mythological and procedural problems of
conducting unobtrusive electronic evaluations of services were uncovered.
General observations on the quality of electronic reference services and
suggestions for additional research are offered.
Background and Setting
Previous research into the quality
of online real time library reference services was not located by conducting
standard literature reviews and by making personal contacts in the library
evaluation research community. However,
there have been a few studies of chat reference services that go beyond just
reporting what types and the frequencies of questions posed. There are at least
two good examples of these efforts. One is the work of Matt
Marsteller and Paul Neuhaus (2001) at Carnegie Mellon University. The Web page
for their paper is:
http://www.contrib.andrew.cmu.edu/~matthewm/ALA_2001_chat.html. Another is by JoAnn Sears (Chat Reference Service: An Analysis of one Semester’s Data) published in Issues in Science and Technology Librarianship (Fall 2001) and the Web page is: http://www.library.ucsb.edu/istl/01-fall/article2.html.
Unobtrusive research into the effectiveness of reference is well documented by Peter Hernon and Charles McClure.1-5 Their findings point to a 55 percent rule. When you ask a question of a library reference service you will get a correct answer 55 times out of 100. The other 45 times the answer will be incorrect.
To see if this 55 percent rule held in the new ‘Chat’ service a pilot project idea was born.
This research effort was a class project for a graduate level seminar in the College of Information Studies at the University of Maryland titled ‘Planning and Evaluation of Library Services’ offered during the Fall of 2001. The goal of the class is to have students learn about and understand different methods used to evaluate library services and to take part in an evaluation study.
An
application was filed and accepted with the University of Maryland’s Human
Subjects Review Committee. This is a
basic step for any research involving human beings and is standard research
protocol. It also provided a
learning tool and a clear articulation of the project’s intent, methodologies,
and risks.
The
services queried were selected from those listed on Gerry McKiernan’s LiveRef(sm) Web page http://www.public.iastate.edu/~CYBERSTACKS/LiveRef.htm that did not openly report that they were
restricted to a specific set of users. Many services limit whom they will serve
for a variety of reasons. At the time the sample was selected (Fall 2001) 36 libraries met this set of
criteria. Six of the libraries were
public and 30 were academic. No library
was queried more than 6 times. Each student investigator was assigned 30
libraries. They each attempted to
conduct 15 chat sessions and they sent 15 emails. Students posed the same
question to each of their assigned services. This provided a good perspective
for comparisons between services (email and chat) and consistency in sampling. The questions asked were of two general
types, ready reference and in-depth or research questions.
The
unit of analysis was the individual chat and email sessions, not the libraries
or the individuals providing the services. There were no plans to compare performance
of libraries or individual service providers.
A much larger sample than six chat and six emails per library service
would be required.
The students and the instructor for
this seminar developed the questions.
The questions were drawn from experience and from real questions posed
at a variety of reference desks. Each
of the questions was researched and three or more basic sources were located
that held the correct answer. The sources were all readily available. They were
governmental and professional association Web sites, basic ready reference
resources such as almanacs, statistical abstracts and other commonly held
reference tools. None of the questions were deemed to be “stumpers” or
difficult to understand. However, the questions required some work to answer
correctly and completely. The questions
used are displayed below in Figure 1.
Time available to collect data and the sample size are key limitations to the generalizability of the findings. The student investigators, many of which work full time or part time in addition to their courses at the University, had limited time in which to collect data. The time they had to collect the data did not always match the hours the chat services were open. The email questions were not affected as they could be sent any time. The timing of data collection, late in the term, did not make the task any easier. The number of services that were available was also limited and we did not want to pose too many questions to any one service. We did not want to create an undue burden on these services. As it was, we detected on at least two occasions that we were asking the same person the same question. However, the librarian or service provider was working for a different library each time. One or both of the libraries may have been contracting out the chat service during the times we used these two services.
The useable samples numbered 133 for the chat services and 107 for the email services. Clearly these samples are not large enough from which to generalize. Therefore, the results of this pilot study should not be taken as representative of the services provided at the time of the sample. However, the sample does give us some insight into the services being offered during late Fall of 2001. Many methodological lessons were learned from this effort and will be used in future research that will be done on a much larger scale.
A number of ethical issues emerged during the development
stages of this project. These were discussed extensively in the seminar. The
first concern was the basic nature of an unobtrusive study. The issue being, is
it fair to study people and services when they are unaware of the
evaluation? It was pointed out in the
class discussions that unobtrusive evaluation studies are conducted all the
time. There are “mystery shoppers” and
the monitoring of services offered via one-way mirrors are conducted daily.
Another issue was the possible waste of library resources. That librarians were extending services to
people who did not have real questions was the issue. This was seen as being balanced by providing the community with a
full report of the findings so adjustments in services can be made where needed
and desired and research methodologies refined. Another issue was the
confidentiality of the subjects, both the librarians and the libraries studied.
This concern was addressed by strict adherence to the recording and reporting
protocols. The names of all librarians
were blocked out by the individual student investigators and never reported to the principal
investigators. The analysis and reporting of the answers to questions was only
done on an aggregate level. Individual
libraries were never analyzed and code numbers were used for each institution.
Institution’s names were not used. The final issue was the misrepresentation of
oneself when asking questions. Student
investigators did not want to be dishonest. So when asked for their status they
would report ‘graduate student’ and if asked for a zip code they responded with
their zip code for their homes. The students used their personal email
accounts, not their university accounts when asked for an email address. A number of chat sessions were completed by
email messages. The acceptance of the research protocol by the campus Human
Subjects Review Committee’s approval of the project helped to address these
ethical concerns. Unobtrusive research
clearly brings into the research process an additional set of ethical issues.
Data Analysis and Findings
The data was collected by the
individual student investigators and recorded in a standard format. A copy of
the general formats used for the chat and email data collection are shown in
Appendix A. This data was then keyed into a spreadsheet and the summary data
was displayed and general calculations were generated. The main summary (see Figure 2. Percentages
for Chat and Email Sessions) displays for both chat and email for each question
the number of useable sessions, number and percentage correct, if references
were cited and the percentages, and the average time. The data shows a great range between the twelve questions posed
and to a lesser degree between chat and email. The chat sessions range between 0.0% and 84.6% correct and email between
14.3% and 100.0%. It is clear that some of the questions were more difficult
than others. This does raise the possibility that we posed some very difficult
questions or some of those sampled in the pilot study just had a difficult time
with some questions.
The duration of the chat and email transactions vary greatly. The average time for each, chat in minutes and email in hours, are displayed in Figure 2. The variation within a single question is also great. On average, it takes less time to provide an incorrect answer than a correct answer for the chat sessions. There is no clear relationship between method of asking a question (email or chat) and the answer being correct or incorrect. The data does show that 8 of the 12 questions were answered on average better via email and 2 of the 12 were less correct via email, and 2 were equal for email and chat.
It is very heartening as library administrators and as reference librarians to see that more correct answers were given with cites. It is also logical to see that more incorrect answers were without cites. Figure 3 displays a matrix showing counts and percentages for correct and incorrect answers, with and without cites for both chat and email sessions.
|
Figure 3: Correctness of Answers and
Inclusion of Cites |
|
|||||
|
Conditions |
|
Chat
sessions |
Email sessions |
|||
|
Correct with
Cites |
|
47 |
35.3% |
52 |
48.6% |
|
|
Correct
without Cites |
26 |
19.5% |
12 |
11.2% |
||
|
Not correct
with Cites |
18 |
13.5% |
19 |
17.8% |
||
|
Not correct
without Cites |
42 |
31.6% |
24 |
22.4% |
||
|
Total |
|
|
133 |
|
107 |
|
Turning now to the chat sessions that
were not accomplished we find the key reason was the services were not
available when the student investigator tried to engage them. Part of this was
due to the time the sessions were attempted and to the fact that most of the
services are very limited in the hours they are available. Technical problems were also a factor in
about 5 cases. Other reasons were that
the service was no longer available and that to use the service you needed a
local zip code. Figure 4 displayed below reports all the reasons for not being
able to accomplish the chat sessions. Both overall and internal percentages are
given.
|
Figure 4: Chat Sessions Not Completed |
|
|
|
|||
|
Sessions |
Overall Percentage |
Internal Percentage |
Reasons |
|
|
|
|
1 |
0.56% |
2.13% |
Assumed
I was at their institution, could not use database(s) |
|||
|
3 |
1.67% |
6.38% |
Chat
service no longer available |
|
||
|
1 |
0.56% |
2.13% |
Connected
to same librarian at different library |
|||
|
32 |
17.78% |
68.09% |
Service
not available |
|
|
|
|
3 |
1.67% |
6.38% |
Needed
local zip code to use |
|
||
|
5 |
2.78% |
10.64% |
Technical
problems |
|
|
|
|
1 |
0.56% |
2.13% |
No
one there but service open |
|
||
|
1 |
0.56% |
2.13% |
No
answer given, too busy to help |
|
||
|
|
|
|
|
|
|
|
|
47 |
26.11% |
100.00% |
Total |
|
|
|
|
180 |
|
|
Total
number of chat sessions attempted |
|||
The reasons why 73 of the email
questions were not answered is somewhat puzzling. Some 26% of them were never
provided a response. This is 10.56% of the total emails sent. Is this a clear
lack of manners? We are sure that many users would see it that way. A
surprising finding was that 10% of the chat services did not have a general
library email question address. Not surprisingly, the biggest reason for not
getting answers was not being a student of a given institution. This was true
in 25 cases. Being referred to public libraries, Internet ‘Ask A’ services, or
given only terms to search were among some of the other reasons email questions
were not answered. Figure 5 below reports all the reasons why the email
questions were not answered. Both overall and internal percentages are given.
|
Figure 5: Email Questions Not Answered |
|
|
|
||||
|
Emails |
Overall
Percentage |
Internal
Percentage |
Reasons |
|
|
|
|
|
19 |
10.56% |
26.03% |
Never
provided any response |
|
|||
|
18 |
10.00% |
24.66% |
No
email reference available |
|
|||
|
4 |
2.22% |
5.48% |
Did
not try to answer |
|
|
||
|
25 |
13.89% |
34.25% |
Not
a student there |
|
|
||
|
2 |
1.11% |
2.74% |
Referred
to AskJeeves or a library database |
|
|||
|
2 |
1.11% |
2.74% |
Referred
to public library |
|
|
||
|
1 |
0.56% |
1.37% |
Beyond
the scope of this service |
|
|||
|
1 |
0.56% |
1.37% |
Only
gave terms for searching |
|
|||
|
1 |
0.56% |
1.37% |
Three
responses, some good some bad |
|
|||
|
|
|
|
|
|
|
|
|
|
73 |
40.56% |
100.00% |
Total
emails questions not answered |
|
|||
|
180 |
|
|
Total
number of emails sent |
|
|||
Student
investigators coded the transcripts for their completed chat and email session
to determine if they included the elements of a reference interview. The
elements used are shown below in Figure 6. These elements were taken from a
number of different sources and worked into questions.
Three
key things were learned from this coding activity. First, it is very difficult
to have a group of 12 people code for these elements. Consistency is a real problem.
No reporting at this time of the presence or absence of these elements
in relationship to correct and incorrect answers is possible. The recoding of the full set of data is
needed. Secondly, we as a profession
need to define what is a “good” chat reference interview. It is very clear from
this pilot research effort that the range of answers provided is very
wide. To illustrate this point we have
selected four transcripts of transactions to show as examples. Two good
examples, one chat and one email reference transaction and two poor examples,
one chat and one email reference transaction. These are displayed in Appendices
B, C, D, and E. Third, we also need to
define and apply a set of “Best Practices” for our reference interactions. This
action would clearly raise the level of our performance as an industry.
Working as chat operators ourselves over the past 15 months, in addition to conducting some focus group interviews with other chat operators has taught us a few lessons. One thing we've learned is that answering a chat question is not the same as answering a question "face to face" or over the telephone. During the initial phases of a virtual reference service chat operators have different feelings. While many are completely comfortable with the technology of chat, many suffer from what they call "stage fright." The ring of the bell strikes panic in some, and enthusiasm in others. Many new chat operators complain that the energy spent becoming comfortable with the technology, and with learning to correspond with anonymous customers, overpowers the ability to conduct good reference interviews. Also, many feel the need to "think fast" and sometimes give an incomplete or a rushed answer, while others are so comfortable with using the web and "multi-tasking" that they find they are able to give excellent service and accurate, "on the mark" answers almost every time.
From our experience it seems that we
are offering a valuable service that is appreciated and needed by our
customers. From the customer’s point of view, we are doing a “great job” by
offering them this service. We almost always receive positive comments, such
as, “thanks, this saved time,” or “this is a great service, I will use it
again.” In general, customers seem to be more satisfied with chat than email,
probably because they are able to connect with a “live” person. However, it does
not seem to be a burden on the customers when we need to follow up, or complete
questions with an email response.
Because of the ability to reach remote customers, and the opportunity to
allow them to “connect” with the library, this is a service that has obviously
added value to our existing services.
Conclusions and Suggestions
for Future Research
A number of
conclusions and suggestions for future research can be drawn and offered from
this study, even with its limited sample size. The sample size does preclude
generalizing to the types of chat services studied, mainly academic and some
public. The sampling systematically excluded
‘closed’ chat services, further limiting generalizability. There may
also be many more libraries offering chat services that were not listed in
‘LiveRef’ (such as special libraries).
Therefore, we cannot with any degree of statistical confidence, state
that the 55% rule holds for chat services for the types of libraries studied,
or in general to all types of libraries offering chat services. Future research will need to address these
sampling issues and some other problems unique to the unobtrusive study of chat
services. One of these other issues is
the need to have the questions appear to be coming from a ‘local’ customer. A
local IP address would be one solution. This can be accomplished by finding
library managers and librarians willing to take part in unobtrusive research
efforts of this type by having them provide a local computer access account. Another issue is, who is providing the
service? Is it the staff of a given
library or is it a contractor? In some cases a consortium of libraries may be
providing the service. From a general customer’s point of view however, this makes
no difference. The difference comes from
knowing the level of performance for staff, contractor, or both.
Customers’ reactions to this new
reference service are key areas for new research efforts. The transcripts that
are generated and the knowledge in some cases of who the customer is, provide
rich resources upon which to conduct productive investigations.
This
pilot study did demonstrate that quality chat services (timely, correct answers
to questions with references noted) are being provided by a number of different
libraries. The study also demonstrated that some services are of a very poor
quality. Hopefully through additional
research, staff training and continuing education programs quality chat
services will be offered to all our library customers. Those who believe chat
reference services are a fad or too costly and therefore will go away, are
clearly in error. This can be proven by asking any library customer that just
had their question answered correctly, quickly, with references noted, who did
not need to leave their desk at home or in the office to get their answer.
Library customers like, and soon will expect, to be able to pose their
questions via a personal visit, by phone, by email, and now by chatting over
the Internet.
This project would not have been possible
without the hard work, critical thinking and excellent questions provided by
the following students: Joann Maguire, Elizabeth P. Miller, Lisa Van Milne,
Laura Teresa Mooney, Christina Lynne Pikas, Malissa Ruffner, Peggy Ruppel,
Marla Ann Schlaffer, Juan C. Vega, Tanya J. Whorton, Jennifer Noelle Willard,
and Moira Elizabet Zelechoski. A great deal was learned by all from this very
ambitious class project. These student investigators have made a real
contribution. We also thank Dr. Eileen
G. Abels, Associate Professor, College of Information Studies, University of
Maryland, for visiting the class and critiquing the methodology and for her
assistance with the forms filed with the campus Human Subjects Review
Committee. We also wish to thank Karen
E. Patterson (Head, College of Information Studies Library) for her personal
assistance to students and faculty.
Contact information
Neal K. Kaske,
Ph.D.
Head,
Engineering & Physical Sciences Library, University of Maryland, College
Park, MD 20742-7011. phone: 301.405.9144 fax: 301.405.9164 email: nk20@umail.umd.edu
Julie Arnold,
MLS
Coordinator of
Reference Services, Engineering & Physical Sciences Library, University of
Maryland, College Park, MD 20742-7011. phone: 301.405.9151 fax: 301.405.9164
email:jh276@umail.umd.edu
References
1.
Hernon, P.,
and McClure, C.R. (1983). Improving the Quality of Reference
Service for Government Publications.
Chicago: American Library
Association.
2.
Hernon, P.,
and McClure, C. R. (1986). Unobtrusive reference testing: the 55 percent rule. Library Journal, 111(April 15)37-41.
3.
Hernon, P.,
and McClure, C. R. (1987). Unobtrusive Testing and Library Reference.
Norwood, NJ: Ablex Publishing.
4.
Hernon, P.,
and McClure, C.R. (1987). Quality of data issues in unobtrusive
testing of library reference service:
Recommendations and strategies. Library and Information Science Research,
9(2), 77-93.
5.
McClure,
C.R. and Hernon P. (1987). Unobtrusive testing and the role of library
management. Reference Librarian, 18(Summer)71-85.
|
Service
Provider Number |
Researcher/Question
Number |
Correct Yes=1 No=0 |
Citation Yes=1 No=0 |
Why Not
Correct |
Time
(total in minutes) |
Elements
of Ref. Interview |
Comments |
|
Service
Provider Number |
Researcher/Question
Number |
Correct Yes=1 No=0 |
Citation Yes=1 No=0 |
Why Not
Correct |
Time
(total in hours) |
Elements
of Ref. Interview |
Comments |