13062022

Evidence, expertise, and the self-improving school system

Rob Coe used his 2013 inaugural lecture at Durham to survey the evidence on long-run change in the performance of the English school system. He concluded that standards had not improved over the last 30 years.

Recently, Dylan Wiliam tweeted that maybe, just maybe, we are now starting to see sustained improvements in the quality of teaching and learning.[i] At times, I have been tempted by the same thought. Only time (and more data) will tell.

How can we account for the lack of improvement described by Coe? And what would it take to transition from the flatlining system that Coe observed to the self-improving system that everyone hopes for? This blog sets out one useful way of thinking about this.

The Gifts of Athena

Joel Mokyr tackles an analogous problem in his book The Gifts of Athena. How did we move from millennia of zero economic growth prior to the 1800s, to the sustained economic growth experienced since? Mokyr’s final answer doesn’t translate neatly to education. But the conceptual framework he develops is helpful in thinking about the transition to a self-improving school system.

This framework is built on a distinction between two types of knowledge. First, knowledge that, which refers to beliefs about how the world works. For example, hot air rises. These beliefs are either correct or incorrect. An addition to knowledge that would be described as a discovery.

Second, knowledge how, which refers to techniques for getting things done. For example, how to operate a hot air balloon. Rather than correct or incorrect, these techniques are either successful or unsuccessful. An addition to knowledge how would be termed an invention or an innovation.

This distinction will be familiar to many. But Mokyr adds several original insights, illustrated with examples from the history of science:

  1. Knowledge that constrains knowledge how. It is inconceivable, for example, that somebody would know how to build the first steam engine without first knowing that the condensation of steam creates a vacuum. This is not the only thing you would need to know, but you would need to know it.
  2. A single piece of knowledge that can support many different pieces of knowledge how. Before the steam engine was invented, the knowledge that condensation causes a vacuum was used to invent the steam pump.
  3. The knowledge that underpinning some knowledge how may be more or less broad/general. For example ‘water condenses at below 100 degrees centigrade’ is less broad/general than ‘water condenses at below 100 centigrade at sea level and condenses at lower temperatures at higher altitudes’. More broad/general knowledge that makes for more reliable knowledge how e.g. the design of steam engines for operation at different altitudes.
  4. The least broad/general amount of knowledge that which can underpin some knowledge how is simply the statement that ‘x works’. For example, Henry Bessemer discovered his method for making steel (knowledge how) by accident. Only later did chemists come to discover the underlying chemistry: he happened to be using pig iron devoid of phosphorus. All that Bessemer knew was that it worked.
  5. Both knowledge that and knowledge how vary in how accepted they are. At the individual level, this amounts to somebody’s confidence in some claim. At the social level, this amounts to how widely accepted something is. Claims that are hard to verify are less likely to be accepted or will take longer to be accepted. The effect of tobacco smoke on cancer is a tragic example of such a hard-to-verify claim.
  6. When knowledge how is better supported by knowledge that, people are more likely to accept the knowledge how. For example, several surgeons had found that sterilizing medical instruments reduced post-operative infections, but the practice only became widely adopted after scientists later discovered the role of bacteria in the transmission of infection.
  7. The difficulty of getting hold of either knowledge that or knowledge how can be thought of in terms of access costs. Sometimes access costs are financial, such as university tuition fees. Sometimes they are better measured in time, such as the difficulty of sifting through competing arguments and sources of information to reach a conclusion. Either way, access costs impede the spread of knowledge.

Expertise and the flatlining school system

Let’s look at Coe’s flatlining education system (1980-2010) through Mokyr’s eyes.

Experienced teachers in this system have plenty of knowledge how, derived from years of error-prone learning on the job. However, the sum total of knowledge that is not much larger than the sum total of knowledge how. Like Bessemer and his method of producing steel, expert teachers often just know that certain things work.

However, even these experienced teachers find their hard-won knowledge how to be somewhat unreliable. Like the condensing of water, what works seems to vary in subtle ways across contexts. The knowledge that underpinning teachers’ knowledge how is narrow, making it easy to misapply the knowledge how.

The knowledge how gleaned by expert teachers is also hard for others to verify. Knowledge how can and does pass between colleagues in the form of advice. But acceptance of this advice largely depends on trust. The movement of knowledge around the system is therefore limited to social networks, usually within particular schools. In the absence of supporting knowledge that, the costs of verifying expertise among strangers are usually too high.

This process of trust-based learning from colleagues is also error prone, with teachers borrowing both successful and unsuccessful knowledge how. As with smoking, the classroom environment makes it hard to ascertain the consequences of certain actions. Nevertheless, the sharing of successful knowledge how leads to pockets of excellence emerging in particular schools at particular times.

Crucially, every time a teacher retires, they take with them the accumulated knowledge how that they have gleaned from a careers-worth of careful trial-and-error and advice taking. They could try to write it down, but how would anyone beyond their personal network verify whether it was successful knowledge how? Somewhere, a newly qualified teacher takes the place of the retiring teacher and begins the process of learning on the job from square one.

In sum, the difficulty of sharing knowledge means that the system gains knowledge how at the same rate it forgets it. Mokyr’s framework can explain the flatlining school system.

Evidence, expertise, and the self-improving school system

How might the transition to a self-improving school system happen?

Recent improvements in the quality of research mean that knowledge that about teaching and learning is starting to accumulate. Progress is slow but steady in multiple areas: the science of reading, cognitive psychology, large-scale trials of different curricula and pedagogical approaches, quasi-experimental evaluations of e.g., national literacy interventions. Crucially, once gained, this knowledge that is unlikely to be lost. It does not leave the system each time a teacher retires. This allows for cumulative growth in such knowledge.

Like condensing steam creating a vacuum, a single piece of knowledge that can support the development of multiple pieces of knowledge how. For example, knowing that working memory is limited supports the knowledge how integrating labels within a diagram supports learning, and the knowledge how providing worked examples supports learning. This multiplier implies that the frontier of evidence-based practice can at times advance faster than the evidence on which it depends.

Teachers can also use this knowledge that to verify knowledge how. For example, expert teachers have long recognised the value of asking many questions of their pupils. The knowledge that retrieval practice helps solidify learning in long-term memory helps secure wider acceptance and uptake of this good practice. This helps spread successful knowledge how beyond the confines of personal networks, across the wider system. Knowledge that makes knowledge how more sharable.

Increasing the breadth/generality of knowledge that should also accelerate this process by increasing the reliability of knowledge how. For example, our increasingly broad/general knowledge that about how exactly retrieval practice works allows us to use retrieval practice better. More precisely, our knowledge that retrieval practice consolidates memories through reactivation implies the knowledge how that teachers should provide sufficient time for all pupils to reactivate memories between posing a question and taking an answer. Increasing the reliability of knowledge how further enhances its acceptance.

The school system described here accumulates and spreads both knowledge that and knowledge how. Mokyr’s framework can also explain the self-improving school system.

So what? Speeding up the transition…

Mokyr’s framework might also help us speed up the transition to a self-improving school system. Here are three suggestions:

  1. Recent funding for ESRC education research, the Education Endowment Foundation, and the establishment of the new National Institute of Teaching will help further expand our knowledge that. As well as looking for new knowledge that, these funders should commission research aimed at broadening/generalising existing knowledge that. This might require lab experiments designed to directly test theory. This will help make knowledge how more reliable and, in doing so, help it to spread.
  2. Research synthesis should focus on distilling mental models of teaching/learning on the grounds that these have rich implications for knowledge how. This contrasts with simply aggregating effect sizes in meta-analyses, which provides only very narrow know that – ‘it works on average’. Given the importance of context in education, this is unlikely to be useful for an individual teacher. Mental models provide broader and interconnected knowledge that, which supports teachers’ to reason about how to adapt knowledge how for their setting. For some brilliant examples of this outside of education, see this blog.
  3. While we have made considerable advances in sharing knowledge that around the system (research reviews, books, teacher conferences), we are nowhere near as good at sharing knowledge how in a trustworthy way. Copying of practice frequently occurs, but it is highly error prone. A more trustworthy approach might involve identifying the best teachers using value-added data, systematically observing their practice to see how they use evidence-based teaching practices, and then capturing annotated videos of this triangulated knowledge how. This would provide a less error-prone way of sharing the considerable knowledge how that is already present in the school system.

In sum, Mokyr’s framework helps bring into focus three ways in which evidence interacts with expertise to contribute to a self-improving school system: knowledge that helps develop new knowledge how, spread knowledge how around the profession, and make this knowledge how more reliable. Pessimists sometimes fret that evidence constrains teachers’ autonomy, thereby compromising their professionalism. On the contrary, Mokyr’s framework illustrates how knowledge that gives teachers the basis on which to discuss and share their knowledge how. Indeed, the right kind of knowledge that actually creates opportunities for teachers to generate new knowledge how and reason flexibly about how it might need to be adapted for their context. Evidence therefore connects and empowers teachers, rather than constraining them.

You can find me on Twitter @DrSamSims, read a short Q&A with me here, or find my Google Scholar here

Members can read the full article here.

Leave a Comment