# FFT: ***...

Discussion in 'Education news' started by Vince_Ulam, Oct 17, 2015.

1. ### lanokiaStar commenter

Oh don't worry Scintillant you don't have to convince me of the crapness of FFT. Fully onboard with that one.

I've said it before but will say it again, at the risk of being boring...
A sharp distinction should be drawn between the figures themselves and the expectations deduced from these figures,.
The figures merely tell you such facts as "children with the same (from a very limited choice) initial characteristics as Jimmy Brown (who is in your current year 3, say), tended to get a level "X" in this summers test.
The interpretation of this fact is, of course, not necessarily that Jimmy Brown should be expected to get a level X in 3 years time. Jimmy has acting on him all sorts of facors that were not taken into account in the "estimate", which in any case might have only a slightly lower probability than a level lower or a level higher.
We all know this. So why the intense criticism of the IMO blameless figures producing machine (FFT) instead of criticism of those who interpret them inappropriately?
Do we have to suppress possibly useful information because it might be interpreted wrongly?

3. ### ScintillantStar commenter

Criticism of both is entirely appropriate, are evidence-based and have been made very clearly. They serve no useful purpose whatsoever and are often used incorrectly to make teachers' lives miserable. The estimates from FFT are useless. If you cannot understand why, and you still consider FFT to be an acceptable tool to use, then so be it.

I suggest reading this again http://icingonthecakeblog.weebly.com/blog/fft-tea-leaves-in-education

The secondary estimates are presented in a slightly more palatable version of the older way of presenting estimates still used in primary (a good example of the old secondary version is on David Didau’s Learning Spy blog here), in that the percentages are given as a cumulative possibilities. 41% chance of a B+ looks a bit less appealing when you realise that the model actually means the student is most likely to get a C.

Either way, this stuff shows you two important things:

In Primary, the ‘Estimated Levels’ tell you nothing.
In Secondary, the ‘Estimated Levels’ tell you nothing.

In case it isn’t obvious why this is the case, I’ll repeat: A student could get literally anything between the lowest and highest level available and not surprise anyone with a vague idea of how grouped data works. You might get a B, then again you might not. You might get level 4, then again you might not. The estimate tells you nothing which you, as a child’s teacher or parent, couldn’t work out for yourself.

There are umpteen other things wrong with this model, but here are a few to start with:

• What data is used to produce the regression models for the estimates? All of it? Complete data points only? Partially complete data?
• Is the data in the model, and therefore each estimate, changed each year that a child is in a key stage? If not, why not? If it is, what does it suggest?
• What exactly is the methodology used to produce this magic?

MamaPyjama and Vince_Ulam like this.
4. ### hermitcrabbeEstablished commenter

OK. Clearly schools are looking for something that "measures" their effectiveness, progress, defines them as a "Good School" This is what the government want. This is what ( it seems ) parents want.
They want a " figure" or a " grade" that assesses the schools performance

Frankly, rightly, or more likely wrongly (and immorally) ALIS and FFT and ALP's and whatever else, have been selling themselves to schools on this basis, knowing darn well its not what their statistics do.

So, what is a good measure?

This is like the argument over guns in the USA.
I find myself arguing the NRA case (unfortunately) "It's not FFT that gets teachers into trouble, it's teachers (SMT) that get teachers into trouble."
Perhaps there is a case for more checks before anyone is allowed to have access to FFT!

But I can't agree with the blogger, who tends towards hyperbole.

"In Primary, the ‘Estimated Levels’ tell you nothing.
In Secondary, the ‘Estimated Levels’ tell you nothing."

This is far too 'over the top'. They can be useful. We constantly refer ourselves to some sort of norm in all aspects of our life. It is human nature to compare ourselves to others. Mothers know when the baby is supposed to say its first word, doctors expect you to take statins at a certain age. Most people go to bed before midnight, etc,,
We are also well -versed in explaining our own digression from the norm.
Sometimes these digressions are justified. If you are 7 foot tall you have a good reason for being 18st in weight.
But we do have the human tendency to justify ourselves when it is not really justifiable. If you are 5 foot tall and 18 st in weight, the temptation to blame it on being 'big-boned' proves irresistible to some.

6. ### ScintillantStar commenter

If people understood it, then they wouldn't use it. Simple.

As I said, the issues with it are explained clearly with examples.

How you perceive it - as comparisons to a norm, is just wrong and a massive oversimplification. You cannot predict for individuals from grouped data.

Interactive reports Here you get into the murky world of ‘Reviewing Past Progress’ and ‘Supporting Target Setting (Estimates)’. ‘Reviewing Past Progress’ borrows the idea of ‘Value Added’ from economics, and, like many Data Disaster proponents, the FFT makes the highly disputed assumption that you can isolate a ‘teacher effect’ or ‘school effect’ from a ‘pupil effect’.

I’ve shown before that most people in schools don’t have the knowledge, skills or understanding to question this assumption, which is entirely unjustified and makes Value Added Not Even Wrong. Suffice to say that it simply makes no sense to assume that a child’s educational development is 100% school and teacher and nothing else, much less to model an individual child's future performance based on the performance of entirely different children in the past, but that’s what happens here.

The "blogger" has done some amazing work on the misuse of data and stats and has even changed the way Ofsted use stats (after pointing out many errors with their data tools - RAISEOnline, Dashboard etc.)

His website is a phenomenal resource for teachers to use to understand just what is being done by people who don't understand what they're doing! I recommend it in the highest possible terms to all teachers

http://icingonthecakeblog.weebly.com/

Compassman likes this.

.
They wouldn't use it in the way most do now

We do it all the time. A prediction is not the outcome - It's the most likely outcome, given a finite information set. Using the FFT estimate for a prediction will give more reasonable likely outcomes than most. It's just a starting point - which will hopefully and very soon be subjected to close scrutiny and reset, as more information about the pupil becomes available, and as we get closer to the end point.
You seem to be thinking of SMT or OFSTED as some sort of fundamentalist sect for which FFT estimates represent the word of an all-knowing God. If that is true then they shouldn't be allowed access, just like you wouldn't allow a mental deficient access to a firearm (except in the USA, of course).

All this stuff on the blog is useful in arguing your case with OFSTED, but it needs tidying up I think. I'd like to see examples of a situation where a school had some apparently damning FFT data but managed to show it was not damning at all by adroit argument.

8. ### MiddlemarchStar commenter

"FFT is just a calculator. Results are fed in one end and estimates come out at the other, based on the quite reasonable assumption that if nothing changes the future will be similar to the past."

Indeed. I recall a smug moment I had when a Local Authority Adviser tried to argue about the validity of FFT targets for a specific year group and picked out 'randomly' four kids, proceeding to indicate their KS2 results and thus their targets for GCSE. I let him go on for a while, then told him that one of his random 4 hadn't attended school for two years and no-one actually knew where she was, one had left us the previous week to go to Australia and one - had died.

petenewton likes this.
9. ### irs1054Star commenter

"How am I to get in?", said Alice
" 'Are you to get in at all?' is the first question you should ask."

Is there any sort of measure that gives a simple indication of something as complex as a school?
Probably not because any such indication would need to be as complex as the organisation it reflects.
Also if we don't really know what is expected from our schools how can we tell when we get it or not?
The other part of the problem is that the grading issue has been hijacked by politico-speak. 'Satisfactory' which is the correct term to use has been replaced by meaningless 'Good' and 'Outstanding'. The way to make all teachers 'outstanding' is to make them wear dayglo jackets.

I think part of the problem is that schools are trying to do too much. For most of the population all the things that are expected of most of the GCSE in the different subjects are really not needed once the people leave school. This wouldn't be so bad if it didn't get in the way of the students learning the stuff they do need.

10. ### ScintillantStar commenter

I think you fundamentally misunderstand the problem

11. ### ScintillantStar commenter

No. SMT and Ofsted are largely just people who don't actually realise what they are doing, nor the limitations of the data they are using.

It really needs to stop.

drek likes this.