, You are here: Wiki-Summaries >> Monitor, Report, Evaluate, Improve  >> Reporting >> Policy/Program Status/Capacity Surveys

This Section: Monitor, Report, Evaluate, Improve (MREI)

Monitor, Report, Evaluate to Improve (MREI)  - Policy, Program, Practices, Capacity Surveys

• Glossary Term (GT)
  
• Encyclopedia Entry (EE)
  
• Handbook Section (HS)
  
• Bibliography/Toolbox (BT)
  
• Additional Resources (AR)

<

>

Externally driven and regularly administered policy, program, practices & capacity surveys are another means by which the reach, quantity, duration, features, accessibility and participation in specific interventions, multi-intervention programs and multi-component approaches can be monitored and reported. Policy & program surveys are different than the voluntary use of self-assessment tools whereby schools, agencies or ministries decide to use a survey for their own improvement planning. Policy and program surveys are also different than ad-hoc global regional or national updates and reports which are done when funding is available. Requiring all entities within an organization to complete the survey, random sampling or efforts to ensure a sufficient number of responses can increase the validity of the survey. Using controls and comparison groups through commissioned independent research increases validity even further but is often beyond the resources of many organizations. Policy and program surveys can report on the status, reach, access/client participation, key features of the intervention, program or approach. Other questions can report on capacities such as assigned coordinators, coordination committees, the use of specific implementation, maintenance and scaling up (IMSS) practices as well as organizational capacities and routines   and the degree to which the policy or program has been integrated within the education or other participating systems. Examples of such surveys include applications to specific interventions such as school feeding, multi-intervention programs on tobacco and multi-component approaches such as Child Friendly Schools or Health Promoting Schools. There is an assumption that the extrinsic motivation of externally driven and administered policy and program surveys will lead to improvements in practice. Consequently, advocates, policy-makers and officials often seek to invert the motto that "what matters is monitored" to "monitor to make it matter" to promote the policies and programs that they are already implementing. There have been few studies describing or evaluating the impact of externally driven reports in motivating or supporting significant improvements in policy, programs or practices.
   
This summary was developed from an ISHN project on Monitoring and Reporting that was done in cooperation with the International Union for Health Promotion and Education, with funding provided by the USA Centers for Disease Control & Prevention) This summary was first posted in June 2012 and revised in May 2021 in May 2021 with support from a project done through Simon Fraser University and UNICEF. Currently it has been posted as as a"revised edition" version. The following individuals or organizations have contributed to the development of this topic. Albert Lee, Christine Beyer, Nancy Hudson, Candace Currie, Vivian Barnekow and Doug McCall. We encourage readers to submit comments or suggested edits by posting a comment on the Mini-blog & Discussion Page for this section or posting a comment below:

Externally driven and regularly administered policy, program, practices & capacity surveys are another means by which the reach, quantity, duration, features, accessibility and participation in specific interventions, multi-intervention programs and multi-component approaches can be monitored and reported. Such surveys are different than the voluntary use of self-assessment tools whereby schools, agencies or ministries decide to use a survey for their own improvement planning. Policy and program surveys are also different than ad-hoc global regional or national updates and reports which are done when funding is available. Requiring all entities within an organization to complete the survey, random sampling or efforts to ensure a sufficient number of responses can increase the validity of the survey. Using controls and comparison groups through commissioned independent research increases validity even further but is often beyond the resources of many organizations. Policy and program surveys can report on the status, reach, access/client participation, key features of the intervention, program or approach. Other questions can report on capacities such as assigned coordinators, coordination committees, the use of specific implementation, maintenance and scaling up (IMSS) practices as well as organizational capacities and routines   and the degree to which the policy or program has been integrated within the education or other participating systems. Examples of such surveys include applications to specific interventions such as school feeding, multi-intervention programs on tobacco and multi-component approaches such as Child Friendly Schools or Health Promoting Schools. There is an assumption that the extrinsic motivation of externally driven and administered policy and program surveys will lead to improvements in practice. Consequently, advocates, policy-makers and officials often seek to invert the motto that "what matters is monitored" hoping to "monitor to make it matter" to promote the policies and programs that they are already implementing. There have been few studies describing or evaluating the impact of externally driven reports in motivating or supporting significant improvements in policy, programs or practices.
Examples of externally driven policy and program surveys at the global, regional national level include:

1. UNESCO conducts a global survey/consultation of education ministries every four years as part of its ongoing monitoring of human rights policies. In 2020, this survey was modified to report on global citizenship and sustainable development as part of the monitoring done for the achievement of Goal #4 (Education) and Target 4.7. Unfortunately, health, life skills and wellness were specifically and deliberately excluded from that survey by the Technical Cooperation Group which is responsible for such monitoring decisions.  Global citizenship education (GCED) and education for sustainable development (ESD) are defined as encompassing most aspects of the social role of schooling, even though UNESCO and the other TCG partners know that these terms are both contested and vaguely understood. The term  "human survival and well-being" is defined only in relation to the health of the planet.  Nevertheless, the survey will provide some information relevant to health and social development policies and programs. For example, the coverage of topics such as bullying, violence, extremism and gender equality in policies, curricula and teacher education may be identified through the survey and collection of documents.   
2. The American School Health Policies and Practices Study (SHPPS) is a national survey periodically conducted by the Centers for Disease Control & Prevention (CDC) to assess school health policies and practices at the state, district, school, and classroom levels.It is likely the longest standing (every six years since 1994) and more comprehensive examples of a national policy & program survey. The early cycles of the survey used a survey and telephone interviews.site visits within the education sector only. The last version in 2016 was administered only to school districts and there are no plans to continue the survey. The WHO Global School Health Policies and Practices Survey is similar to the American SHPPS survey and was also funded by CDC. The WHO survey was piloted in 2017 and has been recently made available to countries. The survey asks principals and head teachers of primary schools to report on the status of topic-specific interventions in five areas; the physical and social environment, health services, nutrition services, health education and physical education. WHO provides technical support and advice including facilitation of countries seeking funding for their surveys. An early adaptation of the American SHPPS study was done in 1999 by the Council of Ministers of Education, Canada with support from Health Canada. The written survey and telephone interviews were conducted with the education and health sectors with respondents from the health and education ministries, local health authorities and school districts, principals and teachers with focus groups done with students and parents. The report also emphasized the shared responsibility of the two sectors for school health promotion and HIV prevention by reporting on all items horizontally across the two systems and vertically within the two systems, thereby providing data on implementation and organizational capacity. This survey has not been repeated.
3. School food & nutrition programs are perhaps the most frequently examined topic among global, regional or national issue-focused surveys. The 2020 Global Child Nutrtion Foundation (GCNF) survey of school meal programs is an excellent example of how a topic-based concern can delve deeply into a program area while maintaining a whole child and inter-sectorial approach. The survey is well-connected to a global network of organizations concerned about school food & nutrition. The survey questions examine the reach, duration, intensity, implementation and maintenance issues related to school meal programs in detail. Policy, contextual, capacity and other factors are examined. The nature of monitoring activities is also reported, The involvement, commitments and needs of the various ministries and agencies is also investigated. Nutrition-related interventions such as instruction, informal education, school gardens and food watser programs are part of the survey. Complementary or related programs such as deworming, sanitation, clean water, growth monitoring are part of the survey. Similarly, the linkages with broader multi-component approaches are examined. Survey questions also ask about funding, governance and the transitions of responsibilities for school meal programs. 
4. The Schools for Health (SHE) network in Europe conducted a mapping project on SH policies and programs in 2020. The summary report provides an analysis of the overall findings. The detailed report tabulates the responses to the questions. The project is linked with several existing SHE activities and tools already produced by the SHE network, including a clearly stated description of the whole school approach, a defined set of indicators, school planning tools, a rapid assessment tool, annual conferences and research network. The project is explicitly intended to enable the SHE network as well as member countries to review and improve their activities. The questions in the survey probe into several implementation, capacity and systems-related issues. These include examination of the roles and backgrounds of the national coordinators, a key topic often missed in other surveys. An important distinction is made between having a clearly defined, national, inter-sectoral SH policy framework which is operationalized via policy requirements, guidelines and specific types of monitoring such as school recognition or accreditation and simply having a number of health promoting activities and programs in schools.  
5. The World Health Organization conducts a regular survey of  Reproductive, Maternal, Newborn, Child,and Adolescent Health Policies. The survey has been administered five times since 2009, with the summary report and the country questionnaire for the 2018 survey now published. Most of the questions focus on specific topics related ti health status, primary health care and treatment services from the health sector. However, the child health and adolescent health sections of the survey do ask about comprehensive policies, coordinated services and quality of care standards for these two age groups. One question asks if there is a national policy and inter-ministry coordination mechanism on early childhood development. Another question asks about integrated child health treatment services on several diseases. A question asks if adolescents are a target population for interventions on several physical health topics. Another question asks if the country has national standards for the delivery of a comprehensive package of services to adolescents. Unfortunately, from a schools perspective, there is no differentiation of these child and adolescent health services to determine if they are school-based or school-linked. Two questions ask about the country has national standards for Health Promoting Schools and if there is monitoring of the implementation of these standards. Respondents to the WHO survey were asked to upload their country policy documents as part of the survey. These documents will eventually be available in a database on the entire survey.        

This summary was initially developed from an ISHN project on Monitoring and Reporting that was done in cooperation with the International Union for Health Promotion and Education, with funding provided by the USA Centers for Disease Control & Prevention) This summary was first posted in June 2012 and revised in May 2021 with support from a project done through Simon Fraser University and UNICEF. Currently it has been posted as as a"revised edition" version. The following individuals or organizations have contributed to the development of this topic. Albert Lee, Christine Beyer, Nancy Hudson, Candace Currie, Vivian Barnekow and Doug McCall. We encourage readers to submit comments or suggested edits by posting a comment on the Mini-blog & Discussion Page for this section or posting a comment below:

Text

This summary was first posted in June 2012. Currently it has been posted as  an "excerpt/adaptation", "first draft" or "revised draft" and "first or revised edition" version. The following individuals or organizations have contributed to the development of this topic.
We encourage readers to submit comments or suggested edits by posting a comment on the Mini-blog & Discussion Page for this section or posting a comment below:
Due to the length of Handbook Sections (similar to a book chapter) prepared for this web site and knowledge exchange program, we post these documents as separate documents. Click on this web link to access the draft or completed version on this topic. Come back to this page to post any comments or suggestions. 

 Bibliography/Toolbox on
Key research, reports and resources on this topic are highlighted below.  Many of the topics in this web site also have extensive bibliographies/toolboxes (BTs) published as separate documents. Click on this web link to access the full version of our Bibliography/Toolbox on this topic. These lists use our outline for these collections that we have developed over several years of curating these materials.

Research	Reports, Resources
Understand the Approach/ Program & Its Effectiveness in Different Contexts on Different Aspects of HPSD, Populations How to Implement, Maintain, Scale Up, Sustain, Build Capacity, Leverage to Support Core Components. How to integrate within Education Systems Use of Ecological/ Systems Approach Research Questions/ KDE Agenda	Understand the Approach/ Program & Its Effectiveness in Different Contexts on Different Aspects of HPSD, Populations How to Implement, Maintain, Scale Up, Sustain, Build Capacity, Leverage to Support Core Components. How to integrate within Education Systems Use of Ecological/ Systems Approach Research Questions/ KDE Agenda

The following additional resources are posted on this web site or published by other credible sources. Please send any suggested additions to i[email protected]

For updates and reader comments on this section, go to our Mini-Blog on Monitor-Report-Evaluate-Improve (MREI)

The number of summaries completed or drafted in this section are listed below)

- Overview

• MREI of What: Student Learning? Youth Behaviours? Health & Development? Status of Policies and Programs?
• Evidence, Reports & Data about the Impact of MREI Systems
• Examples & Models of MREI Systems

- Key Definitions/Descriptions

• Monitoring
  - Out of School/Dropout Data
  - Administrative Data Systems (EMIS)
  - Child & Adolescent Health & Development Status
  - Child & Adolescent Attitudes, Behaviours & Development
  - Teacher Norm-based Assessments of Student Development
  - School Physical Conditions & Resources
  - School Psycho-Socail Environment & Supports
  - Student Incidents, Suspensions & Injury Data
  - Student knowledge, skills, learning (Instruction & Extended Education)
  - Health, Development & Employment of Young Adults
  - Teacher Wellness, Work Lives, Supply, Qualifications, Development, Concerns
  - Education Faculty Programs/Capacity
  - Use, Training of Other Professionals
• Reporting
  - Different Reporting Formats (Progress Reports, Success Stories, Practice Stories, Case & Cohort Studies, Inventories & Intervention Mapping, RCTs and Reviews)
  - Global, Regional, National Updates
  - School Accreditation/Award Programs
  - Policy/Program Surveys
  - Practitioner, Student,Parent or  Community Satisfaction Surveys
  - Consolidated Reports on Children, Adolescents & Young Adults
• Evaluating/Assessing/Analyzing
  - Scheduled Program Evaluations
  - Planned or Post-Hoc Case Studies
  - School Inspections
  - Access, Reach, Response Times of School-based and School-Linked Services
  - Access, Reach, Breadth of Extended Education Activities
  - Student Knowledge, Skills & Attitudes in Health & Life Skills Education (tests and Surveys)
  - Health, Safety & Environmental Conditions in and Near Schools
• Improving
  - Self-Assessment Tools
  - School Recognition Programs
  - Using Student, Parent, Neighbourhood & Teacher Satisfaction Surveys
  - Using School Improvement Planning
  - Using Joint Sector Reviews
  - Using Health, Social, Other Sector Quality Management Systems

- Key Concepts/Insights/Issues

• Competition among Monitors
• Accountability-Purposes, Politics, People, Shared
• Indicators: Valid, Relevant or Convenient?
• Continuous Improvement
• Logic Models: Linear or Complex?
• Data-driven decision-making
• Intrinsic or Extrinsic Motivators within Systems

- How to Implement, Maintain, Scale Up, Sustain, Build Capacity, Adapt to Different Contexts, Leverage to Support Core Components

• Data Sharing Across & Within Systems
• Coordinated Data Collection & Analysis
• Segmenting & Comparing Data by Country & Community Contexts
• Building MREI Capacity in Countries

How to integrate within Education Systems

• Educator Experiences with High-Stakes, Standardized Testing
• Time Frames for Education Change vs Other Sector Programs
• Teacher, Administrator & Other Adopter Concerns

- Implications of Ecological/ Systems Approach

• Shared MREI in Multiple, Complex, Loosely Coupled Systems
• Within Child/Youth Reporting Systems in other Sectors
• Links with Early Childhood Education
• Links with Post-Secondary Education

- Research Questions/Knowledge Development

• Research Programs/Knowledge Centres
• Impact on Student Learning
• Cost Effectiveness, Cost Benefits

• Research Methods & Data Collection Issues