Alaska State Library 2013-2017 Five-Year LSTA Evaluation
Alaska
Evaluation Overview
The evaluation of Alaska’s Five-Year 2013-2017 LSTA plan took place from February to April 2017. Using a mixed-method approach involving advanced statistical analysis of performance data and data collection with major stakeholders involving interviews, focus groups, site visits, and surveys, approximately 261 state library staff, librarians and Governor’s Advisory Council members, and patrons participated in the study. The time periods evaluated were federal fiscal years 2013, 2014, and 2015. The evaluation identified and accomplished all eight goals of the study: 1) Assess the impact that the expenditure of LSTA funds has had on Alaska libraries, library staff, and library patrons; 2) Assess the processes used by the Alaska State Library to implement the goals, objectives, and activities of the LSTA Alaska State Plan 2013-2017; 3) Deliver key findings and recommendations, which will be used in developing the next five-year state plan; 4) Satisfy the evaluation requirement of the federal LSTA program as operated by the Institute of Museum and Library Services (IMLS). IMLS evaluation guidelines also state additional goals for any SLAA evaluation[1]: 5) Highlight effective practices of ASL’s LSTA program; 6) Assess the efficacy in implementing the activities used in advancing state goals; 7) Utilize both statistical and qualitative evaluation methods to assess the efficiency in implementing the activities used in advancing state goals; IMLS evaluation recommendations for 2013-2017 LSTA plans[2], also suggests one additional goal: 8) Identify processes at work in implementing the activities in the plan, including the use of performance-based measurements in planning, policy making and administration.
C-1. Identify how you implemented an independent Five-Year Evaluation using the criteria described in the section of this guidance document called Selection of Evaluators. This LSTA evaluation was rigorous, objective, and conducted by an experienced independent, third-party evaluator. Prior to the start of the evaluation, three guiding documents were created to ensure a valid and reliable process was conducted – Alaska LSTA evaluation plan, evaluation crosswalk, and evaluation logic model. All evaluation protocol including interview and focus group questions, surveys, and site visits were planned, developed, and aligned to the evaluation requirements to ensure the evaluation and its findings were valid and reliable. ASL staff reviewed and approved the plan, crosswalk, logic model, and drafts of all instruments prior to implementation.
C-2. Describe the types of statistical and qualitative methods (including administrative records) used in conducting the Five-Year Evaluation. Assess their validity and reliability. The evaluation used a mixed-method qualitative and quantitative approach. The use of an evaluation plan and evaluation crosswalk helped establish strong internal validity and reliability by ensuring all IMLS evaluation and report guidelines, ASL’s 2013-2017 LSTA goals, and prior recommendations from ASL’s 2008-2012 evaluation were identified, documented, and accounted for in both the design and implementation of the evaluation and all associated instruments and protocol. Qualitative methods included gathering all available SPRs and relevant data, interviews with the State Librarian and LSTA administrator, focus groups with other ASL staff, focus groups with Alaska librarians and patrons, online surveys for staff, librarians, and patrons, and seven site visits including visiting funded projects in the Anchorage area. Quantitative methods included several levels of data analysis.
The preliminary level of analysis used basic descriptive statistics including frequencies, percentages, sums, and means when analyzing SPR data and survey responses. Alaska’s public library statistics over a 10-year period were also analyzed. Analysis of Variance (ANOVA) and correlations will also be used to examine major library trends and identify potential relationships between inputs, outputs, and outcomes.
C-3. Describe the stakeholders involved in the various stages of the Five-Year Evaluation and how you engaged them. The evaluation’s target sampling frame included meeting with state library staff responsible for Alaska’s LSTA program, library recipients of LSTA funding, and Alaska patrons and librarians. The total sample for the evaluation was 261 participants. This included interviews (n=8), focus groups (eight focus groups, n=25), seven site visits spanning five days in Alaska, and three different surveys – librarian and staff survey (n=42), patron library survey (n=186).
Recommendations
Based on the results of the evaluation study the following 12 recommendations are made:
[1] This allows rural libraries and short-staffed (or volunteer only) an opportunity to choose a mini-grant and implement innovative projects without having to write one themselves.
[1] Guidelines for IMLS Grants to States Five-Year Evaluation
[2] LSTA Five-Year Evaluation 2013-2017 IMLS Presentation
The evaluation of Alaska’s Five-Year 2013-2017 LSTA plan took place from February to April 2017. Using a mixed-method approach involving advanced statistical analysis of performance data and data collection with major stakeholders involving interviews, focus groups, site visits, and surveys, approximately 261 state library staff, librarians and Governor’s Advisory Council members, and patrons participated in the study. The time periods evaluated were federal fiscal years 2013, 2014, and 2015. The evaluation identified and accomplished all eight goals of the study: 1) Assess the impact that the expenditure of LSTA funds has had on Alaska libraries, library staff, and library patrons; 2) Assess the processes used by the Alaska State Library to implement the goals, objectives, and activities of the LSTA Alaska State Plan 2013-2017; 3) Deliver key findings and recommendations, which will be used in developing the next five-year state plan; 4) Satisfy the evaluation requirement of the federal LSTA program as operated by the Institute of Museum and Library Services (IMLS). IMLS evaluation guidelines also state additional goals for any SLAA evaluation[1]: 5) Highlight effective practices of ASL’s LSTA program; 6) Assess the efficacy in implementing the activities used in advancing state goals; 7) Utilize both statistical and qualitative evaluation methods to assess the efficiency in implementing the activities used in advancing state goals; IMLS evaluation recommendations for 2013-2017 LSTA plans[2], also suggests one additional goal: 8) Identify processes at work in implementing the activities in the plan, including the use of performance-based measurements in planning, policy making and administration.
C-1. Identify how you implemented an independent Five-Year Evaluation using the criteria described in the section of this guidance document called Selection of Evaluators. This LSTA evaluation was rigorous, objective, and conducted by an experienced independent, third-party evaluator. Prior to the start of the evaluation, three guiding documents were created to ensure a valid and reliable process was conducted – Alaska LSTA evaluation plan, evaluation crosswalk, and evaluation logic model. All evaluation protocol including interview and focus group questions, surveys, and site visits were planned, developed, and aligned to the evaluation requirements to ensure the evaluation and its findings were valid and reliable. ASL staff reviewed and approved the plan, crosswalk, logic model, and drafts of all instruments prior to implementation.
C-2. Describe the types of statistical and qualitative methods (including administrative records) used in conducting the Five-Year Evaluation. Assess their validity and reliability. The evaluation used a mixed-method qualitative and quantitative approach. The use of an evaluation plan and evaluation crosswalk helped establish strong internal validity and reliability by ensuring all IMLS evaluation and report guidelines, ASL’s 2013-2017 LSTA goals, and prior recommendations from ASL’s 2008-2012 evaluation were identified, documented, and accounted for in both the design and implementation of the evaluation and all associated instruments and protocol. Qualitative methods included gathering all available SPRs and relevant data, interviews with the State Librarian and LSTA administrator, focus groups with other ASL staff, focus groups with Alaska librarians and patrons, online surveys for staff, librarians, and patrons, and seven site visits including visiting funded projects in the Anchorage area. Quantitative methods included several levels of data analysis.
The preliminary level of analysis used basic descriptive statistics including frequencies, percentages, sums, and means when analyzing SPR data and survey responses. Alaska’s public library statistics over a 10-year period were also analyzed. Analysis of Variance (ANOVA) and correlations will also be used to examine major library trends and identify potential relationships between inputs, outputs, and outcomes.
C-3. Describe the stakeholders involved in the various stages of the Five-Year Evaluation and how you engaged them. The evaluation’s target sampling frame included meeting with state library staff responsible for Alaska’s LSTA program, library recipients of LSTA funding, and Alaska patrons and librarians. The total sample for the evaluation was 261 participants. This included interviews (n=8), focus groups (eight focus groups, n=25), seven site visits spanning five days in Alaska, and three different surveys – librarian and staff survey (n=42), patron library survey (n=186).
Recommendations
Based on the results of the evaluation study the following 12 recommendations are made:
- Prioritize providing library resources and services that align with Alaska’s unique contemporary issues: workforce development and financial literacy, non-traditional circulation items (e.g. tools, fishing gear, interview clothing, etc.), life skills and suicide prevention, broadband connectivity and digital literacy, climate change, etc.
- Prioritize supporting school librarians and school libraries to support all literacies (e.g. reading, information, and digital) for all k-12 children, many of whom may not be visiting public libraries. Professional development and facilitation of a certificate program were mentioned as high priorities.
- Refine grant application process to ensure CE grants are easier to complete, increase equity between urban and rural library funding allocations, and provide funding priorities and dissemination mini/toolbox grants[1] as future options.
- Use a logic model for current and future planning and formative and summative assessment of LSTA allocations by inputs, outputs, and outcomes to ensure all goals are adequately achieved.
- Continue evolving rigorous Outcomes-Based Evaluation (OBE) process to include all recipients of LSTA funding include staffing positions, which will also help populate the logic model each year.
- Annually review recurring, higher cost projects to ensure high Return-on-Investment (ROI)
- Ensure central core staffing at State Library remains strong even if it means using LSTA funds – librarians rely on the State Library for its assistance, support, and expertise.
- Continue focusing on Goals 1 and 2 and publicize and prioritize Goal 3 for increased allocations in the next five years. Add new goals as deemed necessary.
- Examine the possibility of moving staff currently funded by LSTA back to state funds (e.g. SL consultant and the Talking Book Center, etc.)
- Prioritize IMLS priorities 1 (Expand services for learning and access to information and educational resources in a variety of formats) , Priority 2 (Establish or enhance electronic and other linkages and improved coordination among and between libraries and entities), Priority 3 (Provide training and professional development, including continuing education, to enhance the skills of the current library workforce and leadership), and Priority 8 (Develop library services that provide all users access to information through local, state, regional, national, and international collaborations and networks). See table in appendices.
- Prioritize Focal Areas 2 (Information Access), 1 (Lifelong Learning), 4 (Economic & Employment Development), and 6 (Civic Engagement). See table in appendices.
- Prioritize IMLS Focal Groups: 1) School-aged youth (aged 6-17), 2) Individuals living below the poverty line, 3) Children (aged 0-5), 4) Ethnic or minority populations, 5) Individuals that are unemployed/underemployed, and 6) Families. See table in appendices.
[1] This allows rural libraries and short-staffed (or volunteer only) an opportunity to choose a mini-grant and implement innovative projects without having to write one themselves.
[1] Guidelines for IMLS Grants to States Five-Year Evaluation
[2] LSTA Five-Year Evaluation 2013-2017 IMLS Presentation