We share the view that evaluation is, primarily, a cooperative effort directed at program improvement and, secondarily, conducted for the purpose of providing a critical assessment of what program goals have been attained.
We design evaluations that investigate how and why, as well as whether, a particular program is succeeding or failing in its implementation. This utilization-focused approach allows for ongoing discussion, analysis, and suggestions for change.
The issues and questions that are of central concern to the program determine our data collection methods: what program developers hope to accomplish and what questions are of interest to them. In addition, we are attentive to emergent themes and unanticipated outcomes.
We employ a mixed-methods approach to ensure that our understanding of a particular program is both broad and deep. Historically our methods are rooted in qualitative, social science research traditions, such as direct observation, open-ended and semi-structured interviews, focus groups, surveys, and reviews of program products. We meet clients’ needs for quantitative data by analyzing the extent to which participants or other targeted audiences have achieved the program’s goals.
Before we submit reports to funders we share them in draft form with our clients for review and comment. We report findings to both clients and funders in several ways, as appropriate: interim and final reports, informal memos, presentations, articles, and conference presentations. The domain of PERG's work extends across the United States, and the group has a presence at national and international meetings and conferences.
Follow our work by subscribing to our newsletter.
We invite you to explore our work in recent projects:
A Study of the Endicott College Internship Experience
Lead Evaluator(s): Carol Baldassari and Judah Leblang
Partners: Endicott College Administration
Funder: Endicott College
Abstract: The first phase of our study centered on examining students' open-ended responses to existing data, primarily the open-ended responses on the ERC student surveys, which were filled out after completion of their 120-hour (3 week) internships and the full-semester internships. The focus throughout this initial phase [of our evaluation work] was to determine students’ sense of the ‘value added’ and overall impact of their Endicott internship experiences. Through the ERC surveys and discussions with Endicott staff, (including the Director of Internship, Internship Coordinators, and the Undergraduate Dean/VP), we explored the effects of those experiences on students’ ability to connect and integrate their academic course with ‘real life’ work settings, and the effect of internship placements on student decisions about choice of major and future career decision-making and potential employment.
Key research questions:
- How are students responding to their internship experiences at the freshman, sophomore and senior levels? What value/benefits do students identify at each level?
- What areas of learning and skill development do students cite as outcomes of their internship placements and related experiences? How does this learning impact their choice of major and their perspective on their field of study?
- How/to what extent do student internships complement/build on their areas of academic study?
Overall, a vast majority of students at all levels (100, 200, and semester) found their internships stimulating and engaging. For example, the majority of freshman and sophomore students (approximately 95% of the survey respondents in 2012 and '13) described their internships as valuable*. Similarly, more than 90% of 2012 semester interns reported that they were actively engaged in work related to their majors at their worksites, based on their responses to the ERC survey.
The second phase of our study is currently underway as of Fall 2015. Phase 2 is designed to look at the developmental impact of the Program. Specifically, we are exploring how students' participation in three internships, and related courses and seminars, impact these individuals over time, both in terms of career preparation/readiness and skill development. In this study, we are focused primarily on the cumulative experiences of those who have completed the Internship Program. Our study is focused on two groups: 2015 graduates and recent (2012) alumni.
Family Friendly Campus Toolkit
Funder: US Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE)
Endicott College received a three-year grant from the US Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) to establish a Center for Residential Student Parent Programs. PERG has two roles within this project: to develop and pilot a self-assessment toolkit, and to evaluate all other initiatives and activities of the Center.
Family Friendly Campus Toolkit (FFCT)
PERG is currently developing and piloting a Family Friendly Campus Toolkit self-assessment tool. It will help two- and four-year higher education institutions evaluate and strengthen their services for student parents and their children. Instruments and protocols include: Campus Taskforce Guidelines, Profile of Campus Resources, Self-Assessment Rubric, Institutional Data Collection Guide, Student Parent Survey, and Focus Group protocol. Eight institutional partners are currently utilizing these tools and providing feedback. They will spend most of the year collecting and analyzing data, which they will use to develop a plan for improving their services. PERG will spend the following year revising FFCT and preparing for dissemination.
Evaluation of Center Activities
The Center for Residential Student Parent Programs includes several other initiatives that PERG is evaluating. These include grants for innovations and replication of Endicott’s Keys to Degrees model, Advanced Scholar Grants, a Graduate Research Fellowship, an expansion of the Jeremiah-Endicott services for Endicott Boston student parents, and a website and webinars. The Center aims to develop a vibrant network of practitioners, scholars, institutions, and others involved in efforts to support student parents and their children. It also intends to sustain its work beyond the funding period. PERG will look at the process and extent to which these initiatives are successful and the overall goals are met.
Evaluation of the Jeremiah-Endicott Boston Partnership
Lead Evaluators: Debra Smith, Elizabeth Osche, Joan Karp
Funder: Ascend at the Aspen Institute; Annie E. Casey Foundation
Years: 2014- 2015
Started in 2013, Endicott Boston is an urban campus which primarily offers Associates degree programs and high levels of support to low-income students, many of whom are parents, first generation college students, adult learners, and English language learners. Endicott Boston considers it their mission to provide as much support as possible to help these students succeed in school and to graduate, despite challenging life conditions. Jeremiah Program is a non-profit organization headquartered in Minneapolis, Minnesota, with a target population of single, low-income women with children under the age of 5. The core components of the their model are: Empowerment and life skills training, coaching for career-track college education, quality early education for children, and fostering a supportive community. Their original model includes providing safe and affordable housing for all participants on one campus, although Boston is their first non-residential program. This is also their first program with a sole educational partner. Jeremiah is exploring other adaptations as well, such as partnering with an existing housing community.Through this project, single mom's can start with Jeremiah's Empowerment course, enroll at Endicott, and receive additional supports and benefits as Jeremiah-Endicott students.
PERG is tracking and providing formative feedback on partnership development and program implementation issues as these two partners navigate new ways of enhancing their shared desired outcome—successful college experiences for high needs students with children. PERG has also begun to collect data on the children of Jeremiah-Endicott students, using the nationally standardized developmental screening tool, Ages and Stages, which guides parents to assess their own child's development.
The first phase of PERG's work was funded by Ascend at the Aspen Institute, as part of a seed grant to encourage Endicott College and the Jeremiah Project to explore a new collaboration. The project is currently funded by the Annie E. Casey Foundation.
Arts-Integration and Literacy
New England Arts for Literacy Project
Lead Evaluator: Debra Smith
Associates: Gretchen Porter
Partners: Andover, Quaboag, and Salem school districts, Addison, Springfield, and Peabody Essex Museums, and the Program and Evaluation Research Group at Endicott College (PERG)
Funder: The U.S. Department of Education— Arts in Education Model Development and Dissemination
Years: 2014 – 2018
Abstract: The New England Arts for Literacy (NEAL) is a four-year project that includes a multi-faceted professional development program and ongoing support for participating elementary and middle school teachers focused on improving students' literacy through arts integration. The project supports these teachers through the process of creating, implementing, and documenting inspiring learning experiences for students that promote creativity and critical thinking skills through active engagement with texts and the arts.
The foundation of the New England Arts for Literacy project is built on the Performance Cycle, a framework developed by the ArtsLiteracy Project based at Brown University, and now at Habla in Merida, Mexico. The Performance Cycle Framework includes…
Teachers are immersed in the Performance Cycle during a weeklong intensive summer institute. During the school year, they are supported by the project while they design, implement and document curriculum units that use arts integration to develop their students' literacy skills. These units are shared on the project web site and at a spring symposium. The units include collaborative learning activities with museum partners and visiting artists. Teachers may choose to earn graduate credits from Endicott College for their participation.
The trajectory of the NEAL project begins with an initial planning year, followed by the first year of implementation with 40 teachers in five middle schools and four elementary schools in three participating public school districts in Massachusetts (Andover, Quaboag, and Salem). By the final and fourth year, a total of 12 schools across these districts will include approximately 120 teachers that have been trained in the Performance Cycle Framework. The three museum partners support the teachers by hosting NEAL professional development events, providing resources for teachers, and supporting student field visits to the museum. In addition, the project employs several professional development staff and partners, and will develop relationships with visiting artists as it progresses.
The project's desired outcomes include: teachers who can design and facilitate powerful arts-integrated learning experiences that strengthen students' literacy created and shared with peers; schools/classrooms with a positive climate and the capacity to offer high-quality integrated arts units of study; and students who are engaged in learning with a high academic self-concept and excellent reading comprehension. Long term, the NEAL project seeks to demonstrate improved academic outcomes as measured by state assessments in literacy; and to develop and sustain a New England network of schools, organizations, educators and teaching artists who are committed to and skilled at creating meaningful arts integrated learning experiences that scaffold students' literacy; and to disseminate best practices and findings.
Research and Evaluation: PERG takes a design-based research and evaluation approach consistent with that advocated by Fishman, Penual, Allen, Cheng, and Sabelli (2014)
- Centers on persistent problems of practice from multiple perspectives
- Commitment to iterative, collaborative design
- Concern with developing theory/ knowledge through systematic inquiry
- Concern with developing capacity for sustaining change in systems
Working collaboratively with project leaders and participants the research and evaluation focuses on:
- Understanding whether and how the project meets its goals.
- Understanding the experiences of, and impact on, teachers and their students.
- Providing ongoing feedback from this learning to help inform refinement of the project design and implementation.
- Ensuring that the focus is on the project, not on individuals.
Assessing Secondary Teachers' Algebraic Habits of Mind
Lead Evaluator: Carol Baldassari
Partners/Clients: Boston University, St. Olaf College, Education Development Center
Funder: NSF DRK-12
Abstract: Mathematicians and mathematics educators from Boston University, St. Olaf College, and Education Development Center are working collaboratively to develop strategies and tools for Assessing Secondary Teachers’ Algebraic Habits of Mind.
Key research question
- What are the mathematical habits of mind that secondary teachers use, how do they use them, and how can they be measured?
Components of their research include: first, defining an initial set of mathematical habits (MHoM0 considered critical components of educators' mathematical knowledge for teaching at the secondary level. The MHoM construct is closely aligned with the Common Core, and especially its Standards for Mathematical Practice. Second, the team is developing two assessment instruments: a validated paper and pencil assessment and an observation protocol that measure teachers' knowledge and classroom use of MHoM. The ultimate goal of the team's work is to understand the connections between secondary teachers’ mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.
Lead Evaluator: Joan Karp
Funder: National Science Foundation
TERC and the Manomet Center for Conservation Sciences have teamed up to create a model for climate change education that creates a partnership between teachers and scientists. The project design hopes to turn participating classrooms into satellite field stations, with students acting as citizen scientists who collect data of real use in scientific research. Ideally, students will also engage with some of the large data sets they have contributed to. The project is also conducting research on critical barriers that often impede moving a model to scale, including transferring the model to different sites, beyond the personal involvement of scientists.
This project does not have an external evaluation component; instead it is relying on Advisory Board members with a variety of expertise. PERG's role as an Advisory Board member is to consult with the PI quarterly. During these conversations, the PERG evaluator checks on project progress, the status of ongoing issues, and any new issues. This gives the project PI a chance to reflect on important topics, and to hear feedback from an independent source. PERG is also assisting the PI to structure the annual Advisory Board meeting.
Stemlims: Investigating Stem Literacies in Maker Spaces
Lead Evaluator(s): Judah Leblang
Associate: Gretchen Porter
Partners: TERC, Tufts University, Artisan’s Asylum, DGF Technologies, Sprout, South End Technical Center, Cambridge Friends School, Somerville STEAM Academy
Funder: National Science Foundation
Abstract: Investigating STEM literacies in Maker Spaces is a three-year NSF-funded project, a collaboration between TERC, Tufts University and several other local organizations, including the Artisan’s Asylum in Somerville and the Cambridge Friends School. PERG is serving as the project evaluator, looking at how STEMLiMS staff implement their research, which focuses on how experienced 'makers' carry out their work and use "representational practices" (working between various media such as written, graphic and 3-D representations) in the process. The second phase of the research will focus on how these practices can be implemented with school-age youth in both formal (school-based) and informal settings.
As noted above, the first year of the project focused on expert makers and their practices. The research team conducted observations at Artisan’s Asylum, a large maker space/community, and at DGF Technologies, a small start-up that grew out of work conducted at Artisan’s. STEMLiMS researchers observed makers in action at both sites and interviewed them, with a focus on identification of key skills, attributes and values of the makers and the maker communities.
The three primary goals of STEMLiMS as outlined in project documents are:
- Systematically document and describe how STEM literacy practices manifest in maker spaces at experienced and beginner levels of activity;
- Design, develop, implement and refine literacy supports for maker spaces in two schools through rapid iterative design;
- Develop design principles for supporting STEM literacy spaces that can be adapted to local informal and formal maker space learning environments.
Project work consists of three cycles: Cycle One as described above; Cycle Two—with a focus on the practices at two youth-oriented informal education sites (Sprout, in Somerville, MA and The South End Technology Center in Boston), Cycle Three—centered on practices in two school/formal spaces at The Cambridge Friends School and Somerville’s new STEAM Academy. The project researchers plan to produce a series of papers and presentations sharing their findings related to maker spaces and various practices and attributes that encourage the development of science literacy among school-age youth.
PERG’s evaluation approach: PERG evaluators assist the project through review of research activities and related protocols, surveys and other tools; by interviewing project partners, by attending project planning meetings, and by providing ongoing formative feedback to the PI and research team. The evaluators collect qualitative data focusing on the effectiveness of the project team in respect to their key goals; partner perspectives and their involvement in the project; and the overall quality of the research processes implemented by project staff.
Boston Science Partnership
Lead Evaluator: Joan Karp
Associates: Samara Hoyer-Winfield, Elizabeth Osche
Primary partners: University of Massachusetts, Boston (UMB), Northeastern University (NU), Boston Public Schools (BPS)
Funder: National Science Foundation—Math Science Partnership (MSP)
Abstract: The Boston Science Partnership aims to strengthen BPS middle and high school science education, primarily by raising teacher quality, in order to increase student achievement, the number of students taking higher-level science courses, and students entering science, technology, engineering, and math (STEM) higher education programs. Its goals also include improving university-level teaching, training STEM faculty to be knowledgeable partners in science education reform efforts, and creating institutional changes at the universities that will ensure a continuation of support for and involvement with K-12 science education. The project takes a comprehensive approach in order to tackle major institutional barriers to achieving its goals, as well as to test out specific strategies.
The strategies that BSP utilizes to meet project goals are:
- Contextualized Content Courses (CCC)—courses taught by teams of STEM faculty and BPS teacher leaders that provide challenging content courses for science teachers in the context of the Boston Public School’s curriculum and district-approved pedagogical practices
- Collaborative Coaching and Learning in Science (CCLS)—BPS professional learning communities that provide high-quality science professional development within the context of each school
- Vertical Planning (VP)—PD sessions that bring together STEM faculty with K-12 teachers to build a greater understanding and better alignment within the entire science curriculum
- Support for students, teachers, and higher education faculty—this includes an extensive AP support program of activities, STEM seminars on teaching and learning at the university level, focus on issues of tenure and promotion, and inclusion of STEM faculty at community colleges in STEM pipeline efforts
- Partnership between BSP, UMB, and NEU
PERG has taken a 2-pronged approach to the evaluation questions for BSP:
- Project Goals: Has BSP achieved its primary project goals? How? What has been the impact of each of the primary strategies on achieving these goals?
- Project Strategies: What is the design, implementation, quality, contribution of each strategy? What are the benefits, issues, lessons learned, sustainability factors, and impacts?
Report: Boston Science Partnership
Lead Evaluators: Judah Leblang and Joan Karp
Associates: Susan Cohen, Elsa Bailey
Funders: National Science Foundation (NSF) and the National Aeronautics and Space Administration (NASA)
Abstract: Cosmic Questions is an interactive 5000 square-foot exhibition, developed by staff at the Harvard-Smithsonian Center for Astrophysics (CfA), to promote reflection about and interest in “big questions” about the universe and humanity’s place in the cosmos, along with providing the most up-to-date information about the universe. Through their experiences in the exhibit gallery and at a series of related activities—a a short play, a staff demonstration, and a planetarium show—visitors are encouraged to construct meaning and to find relevance in relation to their own world views and life experiences.
Cosmic Questions was framed around three “big questions:”
- “What is the universe like?”
- “Was there a beginning to time?”
- “Where do we fit in?”
In this context, the exhibition had four major goals for its viewers, as described in project literature:
- Learn about key astronomical and scientific concepts, including:
- The composition of the universe and its vast scales of space and time
- The physical and analytical tools of the astronomer; learning from light
- The interplay of models, evidence and explanation in forming our understanding of the universe
- Increase their understanding of the nature of scientific inquiry by engaging in activities that explore “how we know” about the universe.
- Encounter various human perspectives (historical, personal, cultural, artistic, etc.) on age-old cosmic questions.
- Reflect upon their own ideas about the universe and the meaning and relevancy of the ongoing human search for answers to cosmic questions.
Evaluation Activities and methods:
PERG’s evaluation consisted of 3 phases. In 2000, evaluators conducted a series of front-end interviews at Boston’s Museum of Science (MOS), to determine visitors’ interest in and reaction to questions such as: “Are you interested in new discoveries about the universe?” and “Do you have a mental picture of the universe?”
Based on our findings, the exhibit developers created a scaled-down prototype exhibit at the Museum of Science in spring 2001. PERG evaluators collected extensive data at the MOS, and produced a formative report, which proved useful in the creation of the full exhibition.
The summative evaluation of Cosmic Questions was conducted between from September 2002 to June 2003. The evaluators collected data at 2 sites: Boston’s Museum of Science and the Midland Center for the Arts in Midland, Michigan.
Evaluation methods included:
- Exit interviews with museum visitors
- Follow up phone interviews about 1 week after the museum visit
- Survey of visitors who attended the related play “Boy Meets Girl”
- Survey of visitors who attended the related planetarium show
- Interactive observations, in which evaluators accompanied visitors on their trip through the exhibition
Key Findings: Visitors to Cosmic Questions generally found the exhibit highly engaging. They reported learning new information about the universe, and reflected on the ‘big questions’ embedded in Cosmic Questions. Almost two-thirds of our respondents said they had questions stimulated by the exhibit, and more than 60% said Cosmic Questions helped them think about their place in the universe.
Focus on Math: Creating Learning Cultures for High Student Achievement
Lead Evaluator(s): Carol Baldassari and Sabra Lee
Associates: Judah Leblang, Samara Hoyer-Winfield, Elizabeth Osche
Partners: Boston University (BU); the Education Development Center, Inc. (EDC), a non-profit research and development organization; 5 Massachusetts school districts located in the Greater Boston area (Arlington, Chelsea, Lawrence, Waltham, and Watertown); and, as supporting partners, the University of Massachusetts Lowell (UML) and Worcester Polytechnic Institute (WPI) and the Program Evaluation and Research Group (PERG).
Funder: National Science Foundation – Math Science Partnership
Years: 2003- 2009
Abstract: Focus on Mathematics is a 5-year, grades 5–16 Targeted Math-Science Partnership program that offers a multi-faceted professional development program for mathematics teachers in grades 5-12.
Five (5) high schools, 16 middle schools, and 11 elementary schools from the 5 districts are involved in the FoM program. Over 200 mathematics teachers at the middle and high school levels (grades 6–12), as well as some districts’ grades 4 and 5 teachers, participate in FoM professional development opportunities. Together, the middle and high schools in these 5 districts serve almost 15,000 students.
FoM is based on a specific vision/hypothesis that informs the design of their professional development program for teachers, which is that deep immersion in mathematics leads to the development of teachers’ knowledge of mathematics for teaching. The professional development program, called the Professional Academy, was designed to provide a comprehensive set of ‘immersive’ learning opportunities where participants engage in mathematics at a deep conceptual level. Middle and high school teachers from the 5 district partners work alongside mathematicians and mathematics educators in the various professional development activities. The Academy includes a Master of Mathematics for Teaching (MMT) degree and the Professional Development Portfolio—week long summer institutes, school-based teacher study groups led by mathematicians, on-line courses, after-school seminars, and colloquia.
PERG’s evaluation combines qualitative and quantitative data collection methods to assess—
- The quality of the FoM professional development programs for teachers
- The ways in which teachers increased their knowledge of mathematics for teaching, changed their instructional practice, and provided leadership in mathematics;
- The impact of FoM on students in the five districts;
- The impact of FoM on its partners and on mathematicians;
- The institutionalization of FoM approaches and activities.
- In addition, PERG evaluators carried out case study research on 6 teachers who participated in the MMT program.
Reports: Focus on Math
Focus on Mathematics Case Studies
Lead Evaluator(s): Sabra Lee and Carol Baldassari
Associates: Rosalie Torres
Partners: Boston University, EDC, teachers from Chelsea Public Schools, Lawrence Public Schools, Watertown Public Schools, Waltham Public Schools
Funder: National Science Foundation – Math Science Partnership Program, supplemental award
Studies of four middle and high school mathematics teachers that completed Boston University’s Masters of Mathematics for Teaching Program were conducted between 2005 and 2008 to examine whether and how the program contributed to teachers’ deepening their knowledge of mathematics, shifting their instructional practices, and assuming leadership roles in their schools and districts.
Data collection included observations of FoM professional development sessions including those led by the case study teachers; multiple interviews with case study teachers, interviews and surveys at different points in time with FoM program PIs and staff members, Boston University faculty [mathematicians and mathematics educators], district representatives, and the four mathematics teaching fellows participating in the study; classroom observations; and program artifacts.
Reports on the research methodology and one case study have been completed. The writing of the second case study is in progress.
Reports: Focus on Math Case Study Research
Lead Evaluator(s): Carol Baldassari
Associates: Elizabeth Osche
Partners: Gulf of Maine Research Institute
Years: Front-end evaluation: 2004-05
Summative evaluation: 2008-09
LabVenture! at GMRI’s Cohen Center for Interactive Learning
LabVenture! opened its Mystery of the X-Fish in fall 2005 and since then, has invited 5th and 6th grade students throughout Maine to visit the Center free of charge. The ‘mystery’ is based on a keystone species in both the Gulf of Maine foodweb and the state’s coastal economy – the Atlantic Herring. The visiting middle school students use a range of scientific methods and tools to study current, locally relevant research questions. Its ‘hands-on, minds-on’ activities are modeled on actual scientific research being conducted by marine scientists at work along the coast of Maine.
The Center’s LabVenture! program was designed to:
- stimulate student learning about marine science and the Gulf of Maine community
- illuminate current research projects and keep pace with scientific discoveries
- personalize the scientific process
- inspire and enable visitors’ ongoing learning
- generate excitement about science-related careers
After visiting the Center, students have access to personalized websites, with digital scrapbooks of their experiences (microscope images, team photos, video research reports, etc.). Students can also use the site’s online opportunities for self-directed learning and have an active dialogue with GMRI’s research staff. To support teachers, the website provides pre- and post-visit classroom activities.
PERG conducted the front-end and the summative evaluation for the Center. The purpose of the summative evaluation, conducted at the end of 5 years of operation, was to learn the extent to which, and how, teacher and student visits to the Cohen Center influenced learning and teaching both as a result of the Center’s activities at the time of the visit; and post visit. Areas of inquiry included:
- documentation of students’ and teachers’ experiences at the Center, as well as resultant follow-up activities
- evidence that visits had an impact on classroom teaching and learning of science, as well as the science curriculum
- evidence of teachers’ and students’ effort to extend their learning post-visit
- students’ and teachers’ plans for further research in marine sciences
Data collection methods included:
- on-line surveys of all teachers that visited the Center since it opened, as well as a sample of recent student visitors
- site visits to two schools; interviews with teachers and focus group interviews with students who worked together in teams during their visits
- review of student and teacher artifacts
Reports: GMRI Summative
GMRI: Vital Signs
GMRI: VitalVenture: NOAA B-WET program
Infectious Diseases Exhibit at the Marian Koshland Science Museum of the National Academy of Sciences (NAS)
Lead Evaluator: Judah Leblang
Associates: Samara Hoyer-Winfield, Elsa Bailey, Toby Atlas
Funder: National Institute of Health (NIH)
Abstract: The Infectious Diseases exhibit, and its related component on the Koshland Museum website, were funded through a Science Education Partnership Award (SEPA) grant from the National Center for Research Resources, a department of the National Institute of Health (NIH). The primary goals of Infectious Diseases were to:
- Help visitors understand why infectious diseases continue to cause problems and cause challenges for mankind—including the rapid evolution of microorganisms, the overuse/misuse of antibiotics, and issues of land use and economics;
- Help visitors understand what actions they can take to help control the spread of infectious disease;
- Pique visitors’ interest in the topic of infectious disease and stimulate questions.
As outlined in project documents, Infectious Diseases was “intended for non scientists adults in the general public, teachers, school groups at the middle school level and above, journalists, and policy makers.”
Evaluation Activities and methods:
PERG’s evaluation consisted of several phases.
- A series of front-end interviews with visitors to determine their understanding of and interest in the topic of infectious disease
- Exit interviews and interactive observations and follow up phone interviews with visitors at the Koshland Museum
- A series of interviews and a focus group with users, (including general visitors and educators) of the Infectious Disease portion of the Koshland website
- A post-summative-report round of interviews and observations at the Koshland Museum to examine visitors’ reactions to changes/adjustments made to the exhibit
The JASON Learning Program in Rhode Island
Lead Evaluator: Joan Karp
Associates: Gretchen Porter
Partners: JASON Learning Program, Rhode Island Public Schools
Funder: Rhode Island State Legislature
Abstract: The JASON Learning Program in Rhode Island is an inquiry-based science, technology, engineering, and math (STEM) education program funded by the Rhode Island State Legislature. During 2014-15, its third year of operation, it provided free access for 20 districts to the following, depending on cohort:
- Online accounts for all district administrators, educators, and students to the JASON Learning website resources
- Professional development workshops (attended by 84 teachers this past year) and additional teacher resources
- Print resources (student books)
- Admission and on-site programming at the Mystic Aquarium
- Blackstone Valley Field Programs and bussing
- Science Speaker Series events
- Coaches Program for 1-2 teachers from each district
- NGSS Task Force to align JASON Learning materials with NGSS
- Evaluation: The evaluation focused on teacher implementation of and experience with the JASON Learning Program, and on how teachers see JASON Learning as useful for meeting NGSS requirements. Evaluation questions explored the following key topic areas.
- Current year: How have districts, schools, teachers and students participated in the program during the 2014-15 school year? What have been the outcomes of that participation as reported by teachers?
- Next Generation Science Standards: How do teachers see JASON Learning as useful for meeting NGSS requirements?
- Cumulative impacts: How has involvement with the JASON Learning program impacted 4 teachers who have been involved with JASON Learning for at least 2 years?
Data was collected through a survey of 70 participating teachers and through lengthy interviews with four teachers that had taught using JASON Learning materials extensively.
Pakistani Educational and Leadership Institute (PELI) at Plymouth State University (PSU)
Lead Evaluator(s): Judah Leblang
Associates: Samara Hoyer-Winfield
Partners: ITA (Idara-e-Taleem-o-Aagahi, the cooperating agency in Pakistan)
Funder: U.S. Department of State, Bureau of Educational and Cultural Affairs.
The Pakistani Educational and Leadership Institute is funded through a grant provided by the US Department of State, Bureau of Educational and Cultural Affairs, to Plymouth State University in New Hampshire. The project has been funded annually since the 2003-04 school year. A total of approximately 100 individuals have completed the institute over the past five years.
Each year the focus of the institute changes, to meet participants’ needs. The 2008 institute focused on the areas of cultural heritage preservation and environmental stewardship.
The institute is designed to:
- Contribute to the development of the Pakistani educational system (with a primary emphasis on government-run schools), by exposing teacher-trainers, educational administrators and other leaders to new educational methods and ideas.
- The institute also encourages cross-cultural communication and understanding by bringing Pakistanis and Americans together, or as stated in project documents: “to promote goodwill and understanding between the two countries.”
Data collection methods included:
- Interviews with attendees of the PELI institute
- Interviews with the program coordinator at ITA
- Ongoing discussions with the PI/program director at PSU
- Interviews with faculty and staff members
- Examine the PELI blog, surveys and other project artifacts
- Observations of several sessions of the institute
Evolutions After-school Program at Yale Peabody Museum
New Haven, CT
Lead Evaluator: Judah Leblang
Associates: Elizabeth Osche
Funder: National Science Foundation (NSF), Institute of Museum and Library Services (IMLS)
Abstract: The EVOLUTIONS (Evoking Learning and Understanding Through Investigations of the natural sciences) after school program began in 2005 at the Peabody Museum of Natural History at Yale University, and initially involved approximately 40 students. The program was initially targeted toward low-SES/minority students in the New Haven Public Schools. During the first year, the program targeted students in grades 7-12. Currently the program is aimed at high school students.
Annual projects include the mounting of an exhibition and student-produced videos intended to teach science concepts to elementary-aged students. Students also have the opportunity to participate in field trips to regional science centers in addition to an annual college visitation trip.
EVOLUTIONS program foci include:
- Preparing students for post-secondary (college) education;
- Learning about scientific—and other careers;
- Expanding students’ transferable skills for the future;
- Learning about the Peabody Museum and museum careers
In 2008, Peabody staff developed a new program, SCI CORPS, (Science Career Orientation and Readiness Program for Students), which enabled EVOLUTIONS veterans to train to work as interpreters within the Peabody Museum, and to interact with a broad range of museum visitors.
Evaluation Activities and methods included the following:
- Initial strategic planning—serving as a ‘critical friend’ to project staff and development of logic model for EVOLUTIONS
- Ongoing consultation with project director
- Observation of program through two annual site visits
- Numerous focus groups with students in both EVOLUTIONS and SCI CORPS
- Interviews with project director and staff
- Examination of project documents and artifacts
- Interviews with parents of participating students