Arizona State Library 2013-2017 Five-Year Evaluation
Downtown Phoenix
Evaluation Overview
C-1. Identify how you implemented an independent Five-Year Evaluation using the criteria described in the section of this guidance document called Selection of Evaluators. This LSTA evaluation was rigorous, objective, and conducted by an experienced independent, third-party evaluator. Prior to the start of the evaluation, three guiding documents were created to ensure a valid and reliable process was conducted – Arizona LSTA evaluation plan, evaluation crosswalk, and evaluation logic model . All evaluation protocol including interview and focus group questions, surveys, and site visits were planned, developed, and aligned to the evaluation requirements to ensure the evaluation and its findings were valid and reliable. L.A.P.R. reviewed and approved the plan, crosswalk, logic model, and drafts of all instruments prior to implementation.
C-2. Describe the types of statistical and qualitative methods (including administrative records) used in conducting the Five-Year Evaluation. Assess their validity and reliability. The evaluation used a mixed-method qualitative and quantitative approach. The use of an evaluation plan and evaluation crosswalk helped establish strong internal validity and reliability by ensuring all IMLS evaluation and report guidelines, Arizona’s 2013-2017 LSTA goals, and prior recommendations from Arizona’s 2008-2012 evaluation were identified, documented, and accounted for in both the design and implementation of the evaluation and all associated instruments and protocol. Qualitative methods included gathering all available SPRs, Arizona LSTA snapshots, and relevant data, interviews with the State Librarian and LSTA administrator, focus groups with other L.A.P.R. staff, focus groups with Arizona librarians and patrons, online surveys for staff, librarians, and patrons, and two site visits including visiting randomly selected funded projects in Northern, Central, and Southern Arizona. Quantitative methods included several levels of data analysis. The preliminary level of analysis used basic descriptive statistics including frequencies, percentages, sums, and means when analyzing SPR data and survey responses (See Appendix E1). The second level introduced basic correlations (Pearson R coefficient) and analysis of variance (ANOVA) to identify statistically significant relationships and differences in Arizona’s public library statistics over a 10-year period and for counties that have received LSTA funding.
C-3. Describe the stakeholders involved in the various stages of the Five-Year Evaluation and how you engaged them. The evaluation’s target sampling frame included meeting with state library staff responsible for Arizona’s LSTA program, library recipients of LSTA funding, and Arizona patrons and librarians. The total sample for the evaluation was 364 participants. This included interviews (n=5), focus groups (nine focus groups, n=71), two site visits spanning five days in Arizona, and three different surveys – librarian and staff survey (n=140), patron library priorities survey (n=136), and patron information preferences survey (n=58). See Appendix B for full list of all evaluation participants.
Arizona Library Trends: 2006-2015
A 10-year analysis of all public library statistics using analysis of variance (ANOVA) identified a number of statistically significant trends with a p-value or probability that the change found occurred by chance at .05 or below (5% or less chance that the difference found was by chance)[1]. Library trends that increased at statistically significant levels from 2006-2015 include: A 165% increase in state revenue for libraries (p=.05), 41% increase in library website visits (p=.01), 18% increase in state databases (p=.00), 8% increase in total electronic collections (p=.00), 14% increase in total children’s print collection per capita (p=.00), 3% increase in total service hours per year (p=.02), 12% increase in total children’s (ages 0-11) circulation per capita (p=.00), 36% increase in total children’s programs per capita (p=.01), 111% increase in total young adult (12-18) programs per capita (p=.00), 8% increase in children’s program attendance per capita (p=.01), 34% increase in young adult program attendance per capita (p=.00), 242% increase of total computer use by the general public per capita (p=.05), and 3% increase in total user session of public Internet computers per capita (p=.00). Significant library trends that decreased include: a 14% decrease in total print collection (p=.00), 50% decrease in total reference transactions (p=.00), 6% decrease in total circulation per capita (p=.00), a decrease of 62% in total library programs for adults (19+) (p=.00), 71% decrease in adult program attendance (p=.00), and 22% decrease in total library program attendance (p=.04).
A closer look at programs offered and attendance at those programs found that adult programs accounted for 48% of all programs offered followed by children’s programs at 46%, and young adult programs at 6%. In terms of attendance, however, children’s attendance represented 58% percent of total program attendance, adult program attendance was 38%, and young adult program attendance was 4% of total attendance. Children’s programs also had a much higher attendance to program ratio at 28 to 1 compared to adult programs at 17.6 to 1 and young adult programs at 16.2 to 1.
Correlations between all public library inputs and outputs were also calculated seeking to identify any statistically significant relationships. Strong correlations are considered to be in the 0.4 to 0.7 range and anything above a 0.7 range are considered a very strong relationship. Correlations cannot be considered causal relationships but do identify the existence of a consistent, statistically significant positive or negative relationship between variables. Positive (when one increases so does the other) and statistically significant correlations were found between libraries that received an LSTA award and their service population (0.61), number of librarians with an MLS degree (0.499), total FTE librarians (0.49), total FTE staff (0.44), local revenue (0.538), staff expenditures in salaries (0.42), physical collection expenditures (0.489), total electronic collection expenditures (0.65), visits to library website (0.551), total public service hours per year (0.52), total library visits per year (0.46), total physical circulation (0.42), total electronic circulation (0.69), total circulation (0.435), total library programs for children (0.415), total library programs for adults (0.428), total library programs (0.434), adult program attendance (0.427), and total program attendance (0.40).
Some other statistically significant relationships were found between library inputs and outputs including the Percent of Population Registered and city income per capita (0.54), total library visits per capita (0.579), total circulation per capita (0.62), total children’s circulation per capita (0.46), and total computer uses per capita (0.498). The number of staff per capita also has significant relationships with state databases per capita (0.624), total print collection per capita (0.575), total physical collection per capita (0.456), total physical video collection per capita (0.414), total children’s collection per capita (0.482), total public service hours per capita (0.725), total library visits per capita (0.504), total library programs per capita (0.432), total program attendance (0.356), and total computer uses per capita (0.473). Local per capita revenue also had a strong relationship with total circulation per capita (0.57).
Increases in total circulation (all types) per capita also had significant positive relationships with a number of library inputs and outputs including the percent of service population registered (0.624), city income per capita (0.469), total local revenue per capita (0.57), total staff expenditures per capita (0.479), total print collection per capita (0.548), total physical audio collection (0.572), total physical video collection (0.579), total print children’s collection (0.479), total public service hours per capita (0.413), total library visits per capita (0.753), total children’s circulation (0.671), total adult program attendance per capita (0.317), total program attendance (0.308) and total computer uses per capita (0.677).
[1] One Way ANOVA, https://statistics.laerd.com/spss-tutorials/one-way-anova-using-spss-statistics.php
C-1. Identify how you implemented an independent Five-Year Evaluation using the criteria described in the section of this guidance document called Selection of Evaluators. This LSTA evaluation was rigorous, objective, and conducted by an experienced independent, third-party evaluator. Prior to the start of the evaluation, three guiding documents were created to ensure a valid and reliable process was conducted – Arizona LSTA evaluation plan, evaluation crosswalk, and evaluation logic model . All evaluation protocol including interview and focus group questions, surveys, and site visits were planned, developed, and aligned to the evaluation requirements to ensure the evaluation and its findings were valid and reliable. L.A.P.R. reviewed and approved the plan, crosswalk, logic model, and drafts of all instruments prior to implementation.
C-2. Describe the types of statistical and qualitative methods (including administrative records) used in conducting the Five-Year Evaluation. Assess their validity and reliability. The evaluation used a mixed-method qualitative and quantitative approach. The use of an evaluation plan and evaluation crosswalk helped establish strong internal validity and reliability by ensuring all IMLS evaluation and report guidelines, Arizona’s 2013-2017 LSTA goals, and prior recommendations from Arizona’s 2008-2012 evaluation were identified, documented, and accounted for in both the design and implementation of the evaluation and all associated instruments and protocol. Qualitative methods included gathering all available SPRs, Arizona LSTA snapshots, and relevant data, interviews with the State Librarian and LSTA administrator, focus groups with other L.A.P.R. staff, focus groups with Arizona librarians and patrons, online surveys for staff, librarians, and patrons, and two site visits including visiting randomly selected funded projects in Northern, Central, and Southern Arizona. Quantitative methods included several levels of data analysis. The preliminary level of analysis used basic descriptive statistics including frequencies, percentages, sums, and means when analyzing SPR data and survey responses (See Appendix E1). The second level introduced basic correlations (Pearson R coefficient) and analysis of variance (ANOVA) to identify statistically significant relationships and differences in Arizona’s public library statistics over a 10-year period and for counties that have received LSTA funding.
C-3. Describe the stakeholders involved in the various stages of the Five-Year Evaluation and how you engaged them. The evaluation’s target sampling frame included meeting with state library staff responsible for Arizona’s LSTA program, library recipients of LSTA funding, and Arizona patrons and librarians. The total sample for the evaluation was 364 participants. This included interviews (n=5), focus groups (nine focus groups, n=71), two site visits spanning five days in Arizona, and three different surveys – librarian and staff survey (n=140), patron library priorities survey (n=136), and patron information preferences survey (n=58). See Appendix B for full list of all evaluation participants.
Arizona Library Trends: 2006-2015
A 10-year analysis of all public library statistics using analysis of variance (ANOVA) identified a number of statistically significant trends with a p-value or probability that the change found occurred by chance at .05 or below (5% or less chance that the difference found was by chance)[1]. Library trends that increased at statistically significant levels from 2006-2015 include: A 165% increase in state revenue for libraries (p=.05), 41% increase in library website visits (p=.01), 18% increase in state databases (p=.00), 8% increase in total electronic collections (p=.00), 14% increase in total children’s print collection per capita (p=.00), 3% increase in total service hours per year (p=.02), 12% increase in total children’s (ages 0-11) circulation per capita (p=.00), 36% increase in total children’s programs per capita (p=.01), 111% increase in total young adult (12-18) programs per capita (p=.00), 8% increase in children’s program attendance per capita (p=.01), 34% increase in young adult program attendance per capita (p=.00), 242% increase of total computer use by the general public per capita (p=.05), and 3% increase in total user session of public Internet computers per capita (p=.00). Significant library trends that decreased include: a 14% decrease in total print collection (p=.00), 50% decrease in total reference transactions (p=.00), 6% decrease in total circulation per capita (p=.00), a decrease of 62% in total library programs for adults (19+) (p=.00), 71% decrease in adult program attendance (p=.00), and 22% decrease in total library program attendance (p=.04).
A closer look at programs offered and attendance at those programs found that adult programs accounted for 48% of all programs offered followed by children’s programs at 46%, and young adult programs at 6%. In terms of attendance, however, children’s attendance represented 58% percent of total program attendance, adult program attendance was 38%, and young adult program attendance was 4% of total attendance. Children’s programs also had a much higher attendance to program ratio at 28 to 1 compared to adult programs at 17.6 to 1 and young adult programs at 16.2 to 1.
Correlations between all public library inputs and outputs were also calculated seeking to identify any statistically significant relationships. Strong correlations are considered to be in the 0.4 to 0.7 range and anything above a 0.7 range are considered a very strong relationship. Correlations cannot be considered causal relationships but do identify the existence of a consistent, statistically significant positive or negative relationship between variables. Positive (when one increases so does the other) and statistically significant correlations were found between libraries that received an LSTA award and their service population (0.61), number of librarians with an MLS degree (0.499), total FTE librarians (0.49), total FTE staff (0.44), local revenue (0.538), staff expenditures in salaries (0.42), physical collection expenditures (0.489), total electronic collection expenditures (0.65), visits to library website (0.551), total public service hours per year (0.52), total library visits per year (0.46), total physical circulation (0.42), total electronic circulation (0.69), total circulation (0.435), total library programs for children (0.415), total library programs for adults (0.428), total library programs (0.434), adult program attendance (0.427), and total program attendance (0.40).
Some other statistically significant relationships were found between library inputs and outputs including the Percent of Population Registered and city income per capita (0.54), total library visits per capita (0.579), total circulation per capita (0.62), total children’s circulation per capita (0.46), and total computer uses per capita (0.498). The number of staff per capita also has significant relationships with state databases per capita (0.624), total print collection per capita (0.575), total physical collection per capita (0.456), total physical video collection per capita (0.414), total children’s collection per capita (0.482), total public service hours per capita (0.725), total library visits per capita (0.504), total library programs per capita (0.432), total program attendance (0.356), and total computer uses per capita (0.473). Local per capita revenue also had a strong relationship with total circulation per capita (0.57).
Increases in total circulation (all types) per capita also had significant positive relationships with a number of library inputs and outputs including the percent of service population registered (0.624), city income per capita (0.469), total local revenue per capita (0.57), total staff expenditures per capita (0.479), total print collection per capita (0.548), total physical audio collection (0.572), total physical video collection (0.579), total print children’s collection (0.479), total public service hours per capita (0.413), total library visits per capita (0.753), total children’s circulation (0.671), total adult program attendance per capita (0.317), total program attendance (0.308) and total computer uses per capita (0.677).
[1] One Way ANOVA, https://statistics.laerd.com/spss-tutorials/one-way-anova-using-spss-statistics.php