University of Maryland Libraries: Evaluation Plan for Organizational Change

March 25, 2004.

By M. Sue Baughman, Johnnieque "Johnnie" Love, and Maggie Z. Saponaro

I. Introduction

II. Impact of Change in the UM Libraries

• Background
• OD Activities
• Staff Learning Activities
• Personnel Program Activities
• Faculty and Staff Governance Activities

III. Assessment Effort to Date

• Organizational Development
• Staff Learning
• Personnel Programs
• Faculty and Staff Governance

IV. Future Assessment Activities

• Organizational Culture and Diversity Assessment
• Learning Curriculum
• Workforce Planning
• Mentoring
• Faculty Annual Review Oversight Committee
• Shared Leadership Development Proposal

V. Table 1: Summary of Planned Assessment Activities

VI. Conclusion

Appendix 1. Leadership Development in the UM Libraries-Background/Context

Appendix 2. Timeline for Performance, Permanent Status, and Promotion Review

Appendix 3. Mentoring Task Force Report

I. Introduction

“Once, land and capital were the key strategic resources. Now, knowledge is our key strategic resource and learning is our key strategic skill.”
– Al Gore, Lifelong Learning Summit, 1999

The University of Maryland Libraries’ change process began in 1996 and has affected every member of the organization. These changes range from the development of a philosophy for service, to the formation of teams, to the creation of a comprehensive learning program. These changes have placed a greater emphasis on how staff respond to the depth and breadth of customer needs in scholarly communication and information. More importantly, the changes have placed significant emphasis on how staff interact with one another.

Since 1996, several assessment processes have been conducted which have led to the development of additional organizational changes. A comprehensive assessment of the libraries’ progress towards becoming a team based learning organization is critical for a thorough understanding of the extent of the progress made over the past seven years.

In Summer 2003, the offices of Organizational Development, Staff Learning, and Personnel Programs were approached by the Library Executive Council to address the need to systematically develop leadership skills among library staff. These efforts culminated in the development of Working Paper #7: Shared Leadership Development In The UM Libraries. Within this paper, 16 recommendations were outlined, in order to address the need for continual leadership development within the organization. Included among these recommendations was the development of this evaluation plan, which outlines past and future measures for individual improvement as well as organizational change. It is hoped that by implementing a formalized evaluation plan, leadership development efforts as well as other progress can be systematically evaluated in terms of their impact and effectiveness.

UM Libraries’ Organizational Development Goals

The University’s initiatives and the Libraries’ mission and vision focus on quality, excellence, infrastructure and diversity. A number of the Libraries’ specific objectives within the Strategic Plan (April 2001) target an organizational change process through training and skill development, building teams, and support for activities such as recruitment and retention, rewards, and job re-design. The following objectives illustrate these efforts:

• Objective 3. Ensure that the Libraries have, cultivate and retain the human resources essential to a research library of the first rank. (Initiatives One and Two)
• Objective 8. Improve library work climate by addressing the findings and recommendations of recent library surveys and studies. (Initiative Three)
• Objective 9. Diversify workforce, by meeting targeted objectives in an updated library-wide diversity plan. (Initiative Three)
• Objective 11. Promote and recognize contributions by Libraries' staff to University service, professional organizations, scholarly disciplines and to society generally. (Initiative Three)
• Objective 20. Build the Libraries' human resources infrastructure to enable recruiting, retention and first-class support of the University’s academic mission (Initiative Five)
• Objective 20. (C) Build adequate budgets, assignment mechanisms and incentives to recruit reward and retain student assistants, including graduate assistants. (Initiative Five)
• Objective 23. Continue re-design of individual jobs and organizational processes throughout the Libraries for increased flexibility, accountability and performance. (Initiative Five)

Evaluation of these objectives is critical for developing an understanding of where change has occurred and the extent to which that change has had a positive effect for individuals as members of the organization or for the organization itself.

return to top

II. Impact of Change in the UM Libraries

Background

From 1998 to the present, a number of activities have occurred which have framed the organization as we know it today. First among these activities was the creation of the Scholarly Information Service Philosophy by the Services Task Force (1998), which reinforced the guiding principle that services the Libraries provide are user-driven and the “work of every member of the staff contributes directly to the service mission of the Libraries.” Following this statement, the Work Culture Task Force developed a supporting Guiding Statement on Work Culture, which describes the type of work environment desired in the libraries. A listing of 23 “desirable work values” was derived, and the need for the libraries to provide “sufficient opportunities for personal and professional growth” was recognized. By far the most radical changes for the organization were outlined in Working Papers #1 and #2 – issued in 2000 – which outlined the development of teams, and announced the formation of three subject teams (Working Paper #1) as well as programs teams (Working Paper #2).

A comprehensive outline of leadership development and related activities sponsored by the offices of Organizational Development, Staff Learning, and Personnel Programs from 1996 to the present appears as Appendix One to this report.

return to top

Organizational Development Activities

The Libraries’ organizational change process was designed to be a systemic one. UM Libraries initially hired an outside consultant to assist with the development of teams. In 2000 the position of Assistant Dean for Organizational Development was created to provide in-house support for the change process and to provide assistance to teams. The Assistant Dean for Organizational Development provides one-on-one coaching for staff when requested on such issues as feedback, agenda planning and leadership. The Assistant Dean also conducts retreats and facilitates meetings for teams.

Additional activities have occurred that have supported the development process. A Facilitators Team was created to aid groups in team formation, meeting management, decision-making and problem solving. These facilitators have been assigned to work with various teams during the forming stage of their development in order to give attention to the creation of vision, mission and values statements and ground rules for meetings, as well as team building. Additional staff has been recruited to the Facilitators Team due to the increase in requests for facilitators.

Retreat-type meetings and meetings of shorter duration have been held from time to time with staff in leadership positions to define expectations and qualities of leaders. Retreat-type meetings have also been held within Divisions to develop strategic directions for future growth.

return to top

Staff Learning Activities

Since the CQI Task Force report on Staff Training and Development in 1996, and the hiring of the first Manager for Staff Learning and Development in fall 1997, a number of staff training activities have been organized and offered to Library faculty, staff and graduate students. These programs encompass topics related to personal development, computer skills, and professional development. Activities have been designed as a result of a general needs assessment performed in 1998, computer training surveys completed in 2000 and 2002, and supervisory focus groups held in 2003. Results of the 1998 needs assessment and year 2000 Organizational Culture and Diversity Assessment also contributed to the development of the Learning Curriculum, which was formally launched in May 2001.

Overall, attendance at Staff Learning programs is generally limited to less than 50% of the staff, with the exception of “required” sessions such as Meeting Management (Spring 1999, 230 attendees), Electronic Workplace Readiness (Summer 2001, 151 attendees), Corporate Time (Summer 2001, 127 attendees), and the 2002/2003 Customer Service series (Module 1 – 241 attendees, Module 2 – 192 attendees, Module 3 – 185 attendees). Other popular sessions have included the Customer Service Excellence Institute (Spring 2000, 80 attendees), “Learning to Thrive In An Ever-Changing Workplace” (Fall 2001, 95 attendees), and “Leadership and Shared Leadership” (Fall 2001, 86 attendees) sessions.

return to top

Personnel Programs Activities

“Where am I in this change process?” and “What can I do to be successful in this changing environment?” are two major questions that library staff have asked in subtle but sure ways in the last six years.

The 1998 Needs Assessment survey conducted by the Staff Learning and Development Office indicated that a comprehensive orientation program and a mentoring program were needed for new staff. Graduate assistant orientation has been expanded to provide a more complete support system for our GAs. Other issues identified were diversity training, PRD development for staff, and training for supervisors on how to use the PRD process effectively with staff. The issues identified as the greatest needs were supervisory skills, dealing with change, and leadership acuity.

One result from the 2000 Organizational Culture and Diversity Assessment was the creation of the Coordinator of Personnel Programs position. This position, filled in 2001, ensures that the needs of staff are addressed on a consistent basis. Since 2001, several programs have been implemented to meet the needs of library staff including the Telework Program (2002), Support Group for Library Staff (2002), Through the series of training sessions in “Summer of Supervisors,” emphasis has been on developing supervisors as leaders, and thus the development of the Supervisors’ Brown Bag Sessions (2003).

The Coordinator of Personnel Programs serves as ex-Officio on the Diversity Committee, as Faculty Advisor to the Goodwill Committee, and participates on several other committees for faculty performance and permanent status review processes. In addition, the Coordinator provides oversight to the staff performance review management system (PRD).

In April 2000, the University of Maryland Board of Regents established a new personnel policy giving librarians non-tenure track faculty status. As a result of this new program, Personnel Programs coordinates the Permanent Status Review Calendar for library faculty that includes notification and preparation of library faculty for their review process.

return to top

Faculty and Staff Governance Activities

In an article published in Library Matters July 19, 2001, Acting Chair Doug McElrath laid out the plans for the Library Faculty Assembly (LFA). He stated “that it is the self-governance body that gives all of us a greater voice in the direction of the University of Maryland Libraries in the coming years. (p.2) The Plan of Governance details the work of the Assembly. Areas to be developed include establishing policies, services, and programs; moving toward a team-based, learning organization; working on updates to the Libraries’ strategic plan; designing performance review process; creating a mentoring program; and establishing a peer review process for promotion and permanent status.”

Getting organized became the first order of business. Staff elected representatives to serve on the Library Faculty Assembly. Staff representatives have worked cooperatively with faculty in electing officers for the assembly and electing senators to represent the library in the campus governing body. The Assembly held its first meeting on September 7, 2002.

The Assembly includes all faculty and representative staff and students employed by the University of Maryland Libraries. The purpose of the Assembly is to focus on matters primarily pertaining to library faculty status, responsibilities, and development. Examples of the Assembly's work include:

1. discussing and advising on new policies, services, and programs;
2. participating in the continued development of the team-based, learning organization;
3. discussing and advising on updates and revisions of the Strategic Plan;
4. sharing information with and promoting the skills and abilities of the faculty and staff;
5. advising on a mentoring program and creating opportunities for individuals to develop and grow professionally.
6. administering and overseeing appointment, promotion, permanent status, and faculty evaluation processes. (LFA Web Page)

Additional information about the Library Faculty Assembly can be found on: http://www.lib.umd.edu/groups/lfa/lfadetails.html

III. Assessment Efforts To Date

Organizational Development

During the initial formation of subject teams, the Individual-Team-Organization (ITO) Survey was administered in 1998 to gather baseline data on how team members viewed themselves in the team environment, and to generate ideas for team development and training. The survey was repeated with the same group of people in 2000 in order to assess what had changed or stayed the same over the two- year period. The initial results from the 1998 and 2000 surveys contributed to the development of the Learning Curriculum, along with the other data noted in the Staff Learning section below.

Analysis is currently being done for responses to this survey administered in August 2003. The survey group was expanded to include all members of teams in the Collection Management, Public Services and Technical Services Divisions. This sample group of 171 people includes several duplicate participants because of their membership on multiple teams.

Assessments of teams have been conducted using the interview method. Theme analysis of these interviews has been used to benchmark progress of team development. This data analysis identified issues and challenges that have led to the creation of specific strategies to further assist in the team’s development. Some examples of strategies to date include reassignment of responsibilities, training on meeting processes, and using problem solving techniques to resolve issues.

return to top

Staff Learning

As noted by Moore and Seidner (1998), “learning enables change.” Since the implementation of the Learning Curriculum, a number of sessions have been held focusing on leadership skills development. However, although all programs are evaluated through assessment forms, no systematic method has yet been applied for evaluating long-term outcomes of these activities. Assessment of learning outcomes can be viewed in a number of ways. Kirpatrick (1998), developed one such method, and promoted a four-level scheme for program evaluation:

Level 1 – Reaction - “Customer satisfaction”, or “how those who participate in a program react to it….If participants do not react favorably, they probably will not be motivated to learn. Positive reaction may not ensure learning, but negative reaction almost certainly reduces the possibility to learn.” (p. 19-20)

Level 2 – Learning - “the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program.” (p.20)

Level 3 – Behavior – “the extent to which change in behavior has occurred because the participant attended the training program.” (p. 20)

Level 4 – Results – “final results that occurred because the participants attended the program…. Results can include increased production, improved quality, [and] decreased costs.” (p. 23)

Although the “results” aspect of assessment is of interest, it is important to determine what has happened at the other three levels with regards to Learning Curriculum activities. Participation in Learning Curriculum activities is tracked via an online database, but this simply measures “attendance” at a session, not “active learning” on the part of the attendee. The equivalent of “Level 1” has been traditionally measured at each Learning Curriculum activity, via “Program Evaluation” forms submitted by participants and compiled by the Office of Staff Learning and Development. Results of these forms are shared with session facilitators, as a means of immediate feedback. This feedback has been used by facilitators to fine-tune programs, but little else has been done with them. Attempts at gauging programs at “Level 2” have been made sporadically, and are best represented by “longitudinal” surveys completed by individuals attending Meeting Management training (1999) or who participated in all three Customer Service workshops (2003). An additional “post-workshop” survey was distributed in December to supervisors who attended any of the “Summer for Supervisors” sessions in 2003, in order to determine the “value” of these sessions in terms of behaviors and attitudes changed. Despite these assessment activities, there have been to date no true efforts to measure long-term outcomes of programs – whether in terms of behaviors or results. This is an area that needs to be addressed for all Learning Curriculum activities, particularly those that are focused towards leadership development.

return to top

Personnel Programs

For the last two years Personnel Programs has been working to develop and implement programs such as Telework, Orientation, Mentoring and recruitment and retention efforts through the Library Search Committee activity. Greater emphasis is also being placed on Performance Review Management and accountability. Plans are being made to identify and put in place methods of assessment for each of these programs. A comprehensive orientation program is being explored for all student workers.

On December 11, 2003, Personnel Programs and APPSC held its joint annual information session to assist faculty in preparation for Permanent Status Review. Thirteen faculty were in attendance and expressed that they would be making application either for promotion or permanent status and/or both. An evaluation will be requested of each faculty member in attendance.

Librarians at the University of Maryland first achieved faculty status in 2000, so our collective understanding and facility with the system is still under development. By 2002, the Library Faculty Assembly established an Ad-Hoc Committee on Mentoring for New and Junior Faculty. This committee was charged with the task of preparing a comprehensive plan for new and junior faculty to follow that would assist them in successfully meeting the requirements for promotion and permanent status as prescribed in the Implementation Procedures For Board of Regents Policy on Faculty Status for Librarians.

Since mentoring and coaching are proven techniques for providing support for faculty and staff, and developing leaders, much work has been completed to establish a UM Libraries Mentoring Program for faculty and staff. The Ad Hoc Committee on Mentoring for Permanent Status Track Faculty and the Mentoring Task Force made recommendations to the LFA in support of establishing a Mentoring Advisory Committee (MAC). (See Appendix 3) The LFA did not approve the recommendation to establish this Committee in March 2004. However, mentoring of both faculty and staff is a critical issue, which still needs to be addressed.

In the fall of 2002 the Diversity Committee and other interested staff members completed a barometer check of the climate for diversity within the UM Libraries. Twenty-nine attributes were identified and examined as key elements for creating a climate for diversity. All attributes were measured by participants’ perception of what conditions actually exist among staff, in teams, work groups, departments and divisions. Only four attributes, leadership, commitment, open-ended discussions and opportunity for personal development were rated as high, while an overwhelming and significant number of the attributes were rated in the “average” range. Results from the “Barometer Check” gave us clear indication that more work needs to be done in many areas of diversity within the UM Libraries. This activity was done as the Committee began its Work Plan for 2002-2004.

return to top

Faculty and Staff Governance

The importance placed on faculty governance was demonstrated in the elected Chair being given a seat on the Library Executive Council. Previous Chairs and the current Chair were asked to speak to first three years of the Assembly’s existence. Selective comments include:

“2001-2002: Finding Our Voice” by Doug McElrath, First Chair of Library Faculty Assembly

My term as the inaugural chair of the Library Faculty Assembly was primarily characterized by the struggle to invent ourselves as library faculty with a say in the governance of the Libraries. I think it is fair to suggest that most of us did not know what it meant to be faculty, and that my year as chair was focused on urging my colleagues to take responsibility for the LFA, to act and think like faculty. Of course, there were several steps prior to the formation of the LFA that helped us toward this new identity. These included the initial crisis that led to the Board of Regents declaring librarians to be faculty. As we negotiated our concerns with the University and System administration, a number of leaders in the library emerged who recognized that we needed to take charge of our own destinies as professionals. Our peer review system for promotion and permanent status, which predated our faculty status, also gave us a better sense of how we needed to change our sense of accountability. Finally, the group that drafted our first Plan of Governance had to come to terms with providing structure to the concept of shared governance, to create a foundation, which would reflect our own unique library culture within the academic governance framework. It is still a work in progress, but I think there is a critical mass of library faculty members who now regard themselves very differently than they did ten years ago. There have been challenges. For example, I worked to overcome the attitude among some colleagues that the LFA (and its promotion, peer review apparatus) was just an additional burden on already overworked librarians. Also, the formal library leadership represented by the LEC has been uncertain about how equal a partner the LFA is in the formal governance of our organization. These are challenges that I suspect also were part of the experience of my successors as LFA chair.

Comments from Trudi Hahn, Chair, Library Faculty Assembly, 2002-2003

I had several goals for my year as chair, and felt that we made good progress toward all of them, but of course the LFA always will be a "work in progress." My first goal was to establish all the committees set by the original plan of governance and recruit members, as well as to create task forces or ad hoc committees to address issues not spelled out in the Plan of Governance, but which surfaced as we became more aware of who we were and what we needed to accomplish. These issues included mentoring for all staff, contractual faculty, and emeritus and affiliate faculty status. At the beginning of my year about 30 individuals were participating in LFA governance; by the end of the year over 60 different people were involved.

Another of my goals was to provide a forum through the LFA for library staff to enlighten each other about our professional activities, inspiring each other to similar accomplishments and providing a local platform for national presentations. Twelve librarians participated in another important initiative during my year as chair, that I did not even plan for, but which demonstrated the growing strength of the LFA to organize staff concerns, was the spontaneous, concerted response to the proposed cuts in state support for higher education.

As a body, the LFA matured its second official year, learning to work with parliamentary rules of debate and voting, offering motions and amendments in writing, and keeping to a timed agenda. Thus we were able to use our meeting time more efficiently and effectively.

return to top

IV. Future Assessment Activities

OCDA

The original Organizational Culture and Diversity Assessment, administered in Fall 1999, provided a “snapshot” of the status of Library efforts towards achieving the principles of diversity. Results of this assessment included evidence of a need for a mentoring program and development of activities to “develop a sense of community among employees.” Survey data indicated a strong need for managerial training in interpersonal skills, including feedback, team building, meeting management and methods in promoting diversity. Results also indicated a need for improved communication between upper management and employees, as well as standardization of PRD and merit procedures. This assessment will be repeated in 2004, with results of both surveys compared where possible, in order to determine where progress has been made, and where future efforts should be concentrated. Integrating diversity initiatives into the fabric of the UM Libraries is important so that we address all of the issues of a multicultural organization.

return to top

Learning Curriculum

Although all Learning Curriculum activities have utilized an in-session assessment form, and some programs have benefited from information gained in post-workshop surveys, there has been no true effort to date to assess the impact or “transfer of training” that may have occurred as a result of the program. In Spring 2004, three Learning Curriculum activities will be targeted for assessment using Outcomes Based Evaluation methods. As noted by Sheppard:

This system of measuring results replaces the question, ‘What activities did we carry out?’ with the question ‘What changed as a result of our work?’ A focus on measuring outcomes - the effect of an institution's activities and services on the people it serves - rather than on the services themselves (outputs).’

With the assistance of the MIS office, objectives and measurement tools will be developed for the Spring 2004 offerings of Customer Service, Time Management, and Conflict Management programs. These programs were selected for OBE, given the range of individuals they target. MIS and the Staff Education Coordinating Team will target other programs within the Learning Curriculum for OBE as deemed appropriate.

return to top

Workforce Planning

The demographics of the UM Libraries dictate that study and planning take place due to the significant numbers of library faculty and staff who will be retiring in the near future. Over half of our workforce of 219 is fifty years of age and better. This number of potential retirements, along with regular turnovers will mean that the UM Libraries workforce will look very different by 2010. The challenge will be to manage the rate of change, provide cross training and other personnel services as the changes occur. The key will be to maintain the knowledge needed to support the Libraries’ policies, activities and services.

return to top

Mentoring

Despite the creation of a formalized mentoring advisory committee, mentoring relationships will be established on a smaller scale. An assessment process will determine the success of these efforts.

return to top

Faculty Annual Review Oversight Committee (FAROC)

The Faculty Annual Review Oversight Committee has been charged to evaluate the faculty performance review process. The review will identify ways to strengthen the process including recommendations for strategies to review junior faculty and those faculty with permanent status.

return to top

Shared Leadership Development Proposal

Working Paper #7 outlines a wide variety of means by which leadership development can be addressed through the offices of Organizational Development, Personnel Programs, and Staff Learning. Outcomes Based Evaluation methods hold promise for a number of activities outlined within this paper, not just those associated with the Learning Curriculum. For example, the “communities of practice” described within the proposal may also be a candidate for this type of evaluation.

return to top

return to top

Conclusion

Although the University Libraries have made progress towards developing a culture of shared leadership since the advent of teams in 1998, there is still much work that needs to be done in the area of leadership development for individuals within the organization. This evaluation plan provides a framework for a systematic assessment of our efforts. Repeating the Organizational Culture and Diversity Assessment, as well as analyzing the results of the 2003 Individual-Team-Organization Survey will add useful data to our knowledge base. Applying Outcomes Based Evaluation methods to activities such as those sponsored by the Learning Curriculum will also be beneficial. This evaluation plan is a critical element for ensuring that the Libraries’ organizational change process is on track. Evaluation, like learning, is intended to be a never-ending process. In addition, this process is necessary in order for us to better “balance or tradeoff short-term and long-term results and [avoiding] any sacrifice of current performance while investing in capacity-building for the future of the organization” (Sugarman, p. 74-75).

Appendices:

1. Leadership Development in the UM Libraries – Background/Context
2. Timeline for Performance, Permanent Status, and Promotion Review
3. Mentoring Task Force Report and Recommendations

References

Kirkpatrick, Donald. 1998. Evaluating Training Programs: The Four Levels. San Francisco: Berrett Koehler.

Moore, Carol Ann, and Constance J. Seidner. 1998. “Organizational Strategy and Training Evaluation,” In Evaluating Corporate Training: Models and Issues, edited by Stephen M. Brown and Constance J. Seidner. Boston: Kulwer.

Office of Workforce Relations. 2000. A Guide To Strategically Planning Training and Measuring Results. Washington D.C.: U.S. Office of Personnel Management. Available online at: http://www.opm.gov/hrd/lead/Pubs/spguide.pdf Accessed 5 November 2003

Sheppard, Beverly. Showing the Difference We Make: Outcome Evaluation in Libraries and Museums. Institute of Museum and Library Services Website. http://www.imls.gov/grants/current/crnt_obe.htm. Accessed 4 November 2003.

Sugarman, Barry. “The Learning Organization: Implications for Training, ” In Evaluating Corporate Training: Models and Issues, edited by Stephen M. Brown and Constance J. Seidner. Boston: Kulwer.

return to top