Connect with like-minded education professionals and have your say on the issues that matter to you.
Don't forget to look at the how to guide.
Discussion in 'Workplace dilemmas' started by Scintillant, Jan 25, 2016.
Unable to read your link Scintillant, it shows as a virus threat.
That'll be your anti-virus program.
Doesn't open for me, either.
Opens straight away for me.
Fine for me.
Here it is for those who cannot access it
Classroom observation: it’s harder than you think
Professor Robert Coe
We’ve all done it: observed another teacher’s lesson and made a judgement about how effective the teaching was. Instinctively it feels valid. I am a good teacher; I’ll know a good lesson when I see one. We’ve all experienced it from the other side – being observed – but this time the feeling may be more mixed. Sometimes you get real insight from someone who sees what you don’t, questions what you take for granted and makes you think differently. Sometimes they just tell you what they would have done, or focus on some trivial irrelevance.
In September, I gave a talk at the ResearchED2013 conference. Earlier in the day, Tom Bennett had talked about fashions that had been adopted widely, but uncritically, by teachers, such as Brain Gym, VAK, Left/Right-Brainedness. I sensed a hint of smugness among the delegates: they had never embraced such flakiness, or if they had it was short-lived and emphatically renounced. I couldn’t resist trying to challenge any feelings of being ‘holier-than-thou’ by pointing out something they were still doing, but for which evidential justification was no better than the flaky stuff: classroom observation.
The evidence shows that when untrained observers are asked to judge the quality of a lesson, there is likely to be only modest agreement among them. Worse still, even if they do agree that what they see is good practice, it often actually isn’t. I will briefly outline some of this evidence, and then try to explain how something that feels so right can actually be so wrong.
Research Evidence: Can observers spot good teaching?
There are two key issues here. The first concerns the extent to which the judgements made independently by two observers who see the same lesson would agree: in other words, reliability.
Fortunately, a number of research studies have looked at the reliability of classroom observation ratings. For example, the recent Measures of Effective Teaching Project, funded by the Gates Foundation in the US, used five different observation protocols, all supported by a scientific development and validation process, with substantial training for observers, and a test they are required to pass. These are probably the gold standard in observation (see here and here for more details). The reported reliabilities of observation instruments used in the MET study range from 0.24 to 0.68.
One way to understand these values is to estimate the percentage of judgements that would agree if two raters watch the same lesson. Using Ofsted’s categories, if a lesson is judged ‘Outstanding’ by one observer, the probability that a second observer would give a different judgement is between 51% and 78%.
For observations conducted by Ofsted inspectors or professional colleagues, ‘training’ in observation is generally not of the quality and scale used in these studies, and no evidence of reliability is available. Hence, we are probably justified in assuming that the true value will be close to the worst case. In other words, if your lesson is judged ‘Outstanding’, do whatever you can to avoid getting a second opinion: three times out of four you would be downgraded. If your lesson is judged ‘Inadequate’ there is a 90% chance that a second observer would give a different rating.
The second key issue is validity: if you get a high rating, does it really mean you are an effective teacher? Unfortunately, the evidence here is even more worrying.
Strong et al. (2011) used value-added scores to identify ‘effective’ and ‘ineffective’ teachers, showed videos of them teaching to observers and asked them to say which teachers were in which group. In both the experiments where the observers were not trained in observation, the proportion correctly identified by experienced teachers and head teachers was below the 50% that would be expected by pure chance. At this level of accuracy, fewer than 1% of those judged to be ‘Inadequate’ are genuinely inadequate; of those rated ‘Outstanding’, only 4% actually produce outstanding learning gains; overall, 63% of judgements will be wrong.
How can the research evidence be so out of line with our intuition?
The belief that we know good teaching when we see it is so strong that it is a real challenge to be told that research does not support it. Faced with such a conflict, our instinct may be to reject the research. But here are five reasons why our belief may be wrong:
1. Observation produces a strong emotional response
When we sit in a classroom we absolutely know what we like. It is hard not to project our own preferences for particular styles or behaviours onto the situation, and compare what we see with what we think we would have done. Unfortunately, what we like may not correspond well with what helps others to learn.
2. Learning is invisible
Many people claim that you can tell whether children are learning by observing what happens in classrooms. We may think that if we focus on what the students are doing, rather than the teacher’s performance, we will see learning happening. But learning is invisible and if we don’t judge it by teacher behaviours, we have to rely on observable proxies for student learning. I have listed some of these before:
Students are busy: lots of work is done (especially written work)
Students are engaged, interested and motivated
Students are getting attention: feedback, explanations
Classroom is ordered, calm and under control
Curriculum has been ‘covered’ (ie presented to students in some form)
(At least some) students have supplied correct answers (whether or not they really understood them, could reproduce them independently or knew them already).
Of course, they are all related to learning but it is quite possible for any or all of them to be observed without any actual learning taking place.
3. Accepted ‘good practice’ may be more fashionable than effective
If we do look at the teacher, we will try to match what we see with what we know to be good pedagogy. Unfortunately, we are limited here by two things: our ability to define and operationalise specific pedagogic practices, and our knowledge about whether they really are effective. Moreover, if a group of observers are shown the same lesson they might well disagree about whether each of these practices has been seen.
4. We assume that if you can do it you can spot it
For experienced teachers, their classroom behaviour is mostly automated and sub-conscious. If you do not know exactly what it is you do in a way you could describe explicitly, it may be difficult to recognise when you see someone else do the same. Even an effective teacher may not understand fully which bits of their practice really make a difference. And if the observing teacher is experienced but not particularly effective themselves, it may be even less likely they will be able to identify effective practices.
5. We don’t believe observation can miss so much
A number of quite surprising studies in psychology show that when people try to focus on observing particular things, they can miss an extraordinary amount, a phenomenon known as inattentional blindness. For example, in one study around half the observers failed to spot a gorilla walking through the middle of a basketball game when their attention was focused on specific events.
What should we do?
A number of people have written blogs recently about classroom observation, including@joe__kirby, @learningspy, @oldandrewuk, @headguruteacher, @tombennett71,@Cazzypot, @HeyMissSmith, @StuartLock, @Katiesarahlou, @samfr. Various suggestions include that we should disband or reform Ofsted, that lesson observation should be used only formatively, or that we should stop doing it altogether.
My position is this: I am not against Ofsted; I am certainly not against inspection; nor am I against classroom observation. In fact there are good reasons for wanting to observe teaching. It is hard to imagine a credible evaluation system that doesn’t include some observation; when done properly it does contribute to a valid judgement of teaching quality; and the process of having to think harder about how to do it should help us to understand better what effective pedagogy means. However, if we are going to do it we should do it in the most defensible way possible. Specifically, I think we should:
Stop assuming that untrained observers can either make valid judgements or provide feedback that improves anything
Apply a critical research standard and the best existing knowledge to the process of developing, implementing and validating observation protocols
Ensure that good evidence supports any uses or interpretations we make for observations. It follows that appropriate caveats around the limits of such uses should be clearly stated and the use should not go beyond what is justified
Undertake robustly evaluated research to investigate how feedback from lesson observation might be used to improve teaching quality (EEF already has one such study underway).
This certainly does have implications for Ofsted, and I have written and spoken previously about other aspects of the inspection process that ‘require improvement’. However, it also has implications for every school and college leader who wants to use lesson observation as part of an evaluation or quality improvement process.
We all need to raise our game on this one.
Slides from an event organised by Teacher Development Trust and TeachFirst on 13 Jan 2014, “The role of lesson observation in England’s schools”, can be found here. Video of the event can be foundhere.
No problem with the link here. Interesting too.
Point 3 about accepted good practice being more about fashion than learning stood out. I know if I am observed I play to what the observer wants to see. At the same tie some of the lessons pupils consistently tell me are the most useful are the ones where I stand and talk for an hour - that would not be graded well in observation! But the pupils aren't entirely right either. Those lessons wouldn't work as well if they were they only way I ever taught.
Yes, keep 'em guessing. Keep 'em on their toes. Change it up a bit. They've got to be interested.
But that's the whole thing. The ones who sit there looking like congealed rice pudding yet somehow with expressions of contempt on their milk faces quite often end up surprising you.
Oh my good god @Flere-Imsaho what a wonderfully cruel and apt description! It will be in my thought bubble for some of my gang from now on in. The majority of them are ordinary teenagers..enthusiastic today, sulking tomorrow, depending on what way the wind blows..but there are a few eternal contemptuous milk faces
This says it all. (And also made me laugh)
I'm sure this also relates to the known unreliability of 'witness' evidence in court etc.
I show that one to children now and again to make a similar point:
Amazing! (Further discussion would invalidate anyone else's 'result'; even having said 'amazing' probably already has!)
I'm just upping this, as I'd love to see what others thought of @Scintillant 's test.
Edit: Why not put it on Personal?
I think this whole topic of the veracity of 'observed' lessons, is worth a wider audience (?)
I agree. It is very good evidence for teachers to use with ill-informed management. It should inform better practice but will probably be ignored as it makes people admit that what they have been doing has no validity, If they have used it to treat teachers poorly, they will be even less likely to accept the facts.
The video goes into more detail. Every teacher should be fully aware of this area
Christopher Hitchens said 'the mark of incompetence as a leader is ensuring things you want and need from people are never, ever made clear. That way you can forever be pointing fingers at them and telling them they are wrong' OFSTED and the whole teacher observation process in a nutshell.
OFSTED already know all this don't they?
20 people observed a lesson and graded it.
7 gave it Outstanding
3 gave it good
4 gave it RI
6 gave it inadequate
These were all people who had just completed the OFSTED training
Which is why individual lessons should never be graded.
Given that our SMT only exist to assess/discipline/improve us then what else of the incredibly long and vague Teachers' Standards should be used against us?