An Unobtrusive Evaluation of Online Real Time Library Reference Services
by
Neal Kaske and Julie Arnold
for the
Library Research Round Table, American Library Association,
Annual Conference, Atlanta, Georgia,
June 15, 2002
This
empirical study’s main goal was to answer the following three questions. 1) How well is the new ‘Online Live
Reference’ or ‘Chat’ library service performing? 2) Does the 55% rule reported for past research into the level of
correct answers provided by reference services also hold true for this new
service? 3) What can we learn about the reference interview process through
studying the transaction logs for these services? In
order to meet this research goal, a set of 12 questions was developed and posed
unobtrusively to a sample of 36 ‘Chat’ services in late Fall of 2001. The same
questions were also posed to these 36 libraries’ central email services where
available. A total of 180 chat sessions
were attempted and 180 email questions were sent. The chat sessions successfully
completed numbered 133 and the percentage with correct answers was 55%. Email
questions answered totaled 107 and the percentage answered correctly was 60%.
The difference between questions ranged from 0.0% to 84.6% correct for chat
sessions and 14.3% to 100.0% for email.
Many insights into the mythological and procedural problems of
conducting unobtrusive electronic evaluations of services were uncovered.
General observations on the quality of electronic reference services and
suggestions for additional research are offered.