By Afi Y. Wiggins, Managing Director at the Charles A. Dana Center
Across the education spectrum conversations about systemic change center around data and data use. Often, such discussions take place among too few stakeholders and do not extend beyond student outcome data disaggregation. Efforts to transform systems must focus on the way they operate and the decisions and actions systems stakeholders take to promote or impede progress. Disaggregated data on student outcomes is useful, but only to the extent that it provides evidence of how students navigate and benefit or not from education systems. Key efforts to advancing evidence-based solutions that continually improve education systems are:
- Assembling a powerful data and impact team.
- Identifying underlying systemic causes of issues; factors related to structures (e.g., policies, practices, and resources), relationships and connections, and shifts in power dynamics and mindsets[1].
- Identifying relevant data and other information that directly aligns to and informs the underlying systemic causes.
Other than these key efforts to advance evidence-based solutions, systems stakeholders have more options to consider for continuing progress.
Place the onus on the system. Our mindsets influence our actions. There is a difference between asking ‘Why aren’t students passing the course?’ and asking ‘Is the educational environment conducive to student learning?’ The former way of thinking is blameful of the students. The latter way of thinking places the onus on the environment in which students learn. That environment is the responsibility of educators. Believing that every student is capable of learning is a great first step, but it is not enough. System leaders must also internalize and act on responsibility to create conditions of access, opportunity and success so that students may thrive.
When consuming data and information, always ask whether the data informs the issue and, if so, why the results are as they are. Ask these questions whether the results are positive or negative. In asking why, center curiosity on systems, processes and actions people in power take or do not take that may have led to the results.
Establish a functional and dynamic data team. Creating a data team that includes people with varying experiences, perspectives and roles is the first step. Data teams like these ensure that the right data is collected and used, as well as the right decisions be made. Everyone on the team, no matter their power and agency outside the team, should have equal power and agency inside the team. Develop data teams with the following stakeholders:
- Leader – Select someone in a leadership role who can make or influence decisions about priorities and budgets.
- Institutional researcher (IR) or other formal data persons – Working with data people at your institution should be collaborative instead of transactional. Institutional researchers are technical experts — they know databases and statistics. They may not always know the nuances of the how the education system and programs operate. Working collaboratively allows IR staff to inform the team of the available data and what is possible to extract. This collaboration also allows others on the team to provide context for the data that needs to be extracted and how it should be formatted for practical use.
- Evaluators – Individuals with evaluation specialties are experts at surfacing the right inquiries and establishing investigation processes. These evaluation experts can probe to ensure goals and outcomes are linked to relevant data sources.
- Faculty and advisors – The individuals who have the closest touch points to the students should always have a place on the data team. Consider including faculty who are skeptics about the programs or processes the institution implements. Skeptics oftentimes offer perspectives that lead to deeper, more nuanced inquiries. Given the volume of students advisors engage with, they offer perspectives on their roles in guiding students through the education system.
- Students from priority populations – Students typically provide useful insight into their and their peers’ educational experiences. They can offer a different way to conceptualize a problem, identify systemic causes others on the team do not consider, and make sense of findings from data in different ways.
Set common standards of excellence. Student experiences are not just about whether they pass a course, persist and earn a credential or degree. Their education journeys must be analyzed to determine where the systems promote or impede progress. When defining student outcomes, start by categorizing outcomes by access, opportunity and success[2]. Set common standards of excellence that apply to each student regardless of their demographic descriptors[3]. Avoid using data from other student groups to set the bar. Set the bar and apply it to each student group.
- Opportunity – Opportunity is about creating favorable circumstances and conditions for students to thrive. These circumstances include developing high-quality curriculum, educators, learning environments and smooth processes and transitions through systems. When thinking about opportunity, consider the extent to which structures have been implemented that provide the best learning environments for students.
- Access – Access is about allowing students space to enter or experience opportunities. Just because a condition or structure has been created, it does not mean each student has access to that condition. Granting students access must be just as intentional as creating and implementing opportunities. When thinking about access metrics, think about who — by student group — is afforded access to a school or program or course. Determine an overall proportion of students that will be granted access to such experiences and set that exact proportion as the goal or standard for each student group.
- Success – Success must be defined based on a common definition of excellence that will be applied across all student groups. No matter how far away a particular student group is from achieving the common standard of excellence, the standard should not be reduced. Set the expectation at excellence and focus attention on tailoring practices, supports and resources that may be implemented to get students to achieve the standard rather than lowering the standard.
Setting common standards of excellence might mean 80% of students, per student group, enrolled in a math corequisite math course and 80% of the students enrolled pass the course with a B or above within one year. The point here is that the standard should be excellence, not mediocrity, and that the standard should apply to each student, no matter their background or how insurmountable it might seem to move students to that standard.
Examine structural change and student outcome metrics together. Examining policies, actions and resource allocations alongside disaggregated student outcome data provides a comprehensive picture of how education systems operate. The policies, practices and resources inform current conditions and allow stakeholders to envision and improve those conditions. Student data takes the temperature of the system, allowing determinations of what works, for whom and who is benefiting (or not benefiting) from the system.
Data is only useful if it is accurate and informs solutions to issues of systemic change. Simply having access to disaggregated student outcome data is not sufficient. Knowing that certain student groups score above or below other student groups or enroll in courses at higher or lower rates than others is only helpful if that knowledge is used to probe deeper and inform changes to the system.
- Establish and engage in ongoing pragmatic processes & routines to continually improve systems. Evaluators are a great resource for establishing continuous improvement practices and routines. Evaluation practices are typically guided by improvement science[4]. The data team should engage in the following cycles, documenting decisions and actions along the way. Pre-planning – Establish the data team.
- Plan – Establish priorities. Identify problems or questions. Brainstorm and plan solutions.
- Do – Determine what the institution will do. The “do” might be to implement and test an intervention or to investigate a problem more deeply.
- Implement & Test – If an intervention is the “do,” work with evaluators to plan the evaluation of the intervention. Be sure evaluators attend to implementation fidelity or the extent to which the program, strategy or intervention is implemented in the ways it was intended. Consider whether the intervention will be implemented institution-wide or a pilot program.
- Act – Decide on spreading or replicating the intervention, make the intervention permanent (through policy and practice changes), or act on the findings from non-intervention studies.
- Repeat – Return pre-planning or planning.
These actions may seem cumbersome and time-consuming at first. With continuous engagement, they will become normative. The continuous improvement process rarely operates in a linear fashion. Leave room for processing feedback and making decisions to change direction or continue course.
Systemic change is about shifting the conditions that hold a problem in place[5]. Focus attention on improving the system and the decisions and actions of powerful stakeholders in the system. Systemic change takes time, curiosity, care and thoughtfulness. It also takes innovation and bravery. There may be grand failures, but the successes will be even grander. Ultimately, when systems are transformed to ensure every student will succeed, every student will succeed. In doing the work to reform systems, positive student outcomes will follow.
[1] Kania, J., Kramer, M. & Senge, P. (2018). The waters of systems change. FSG Reimagining Systems Change.
[2] Bensimon, E. M., (2009). Accountability for equitable outcomes in higher education. A brief written for the Roundtable on Increasing the Success of Newly Emerging Majority College Students. University of Minnesota: Center for School Change, Hubert H. Humphrey Institute of Public Affairs.
[3] Gutiérrez, R. (2013). The sociopolitical turn in mathematics education. Journal of Research in Mathematics Education, 44(1), 37-68.
[4] https://www.carnegiefoundation.org/blog/improvement-discipline-in-practice
