Rimmer: "But that was a bar room brawl, that was a common pub fight,a shambolic drunken set-to."
Lister: "...which you started. "
Rimmer: "I just made an innocuous comment. I merely voiced the rumour that McWilliams was sexually tilted in favour of sleeping with the dead. I didn't start the rumour; I merely voiced it."
Lister: "...to his face -- right to his face...when he was with his four biggest mates. And then you do your roadrunner act and leave *ME* to face the music.
Rimmer: "Well, I could have got hurt!"
Lister: "You'd have made a brilliant general, wouldn't you?"
Red Dwarf
We all have opinions about what makes for “a brilliant general”. When the New Zealand Ministry of Education's ’next Best Evidence Synthesis claims to have identified “leadership approaches that really have an impact” with respect to the “achievement and well being of students” it behoves us to sit up and take notice.
When I sat up I was dead chuffed to find Phil Rosenzweig’s The Halo Effect ... and the Eight other Business Delusions That Deceive Managers sitting alongside me.
Every school leader believes they can make a positive difference to the achievement and wellbeing of students. Now a Ministry of Education report not only confirms that is the case, but pinpoints the leadership approaches that really have an impact. Educational Gazette Vol 86 Number 11 2007
What makes a positive difference to the achievement and well being of students is the business equivalent of What leads to high performance? – Rosenzweig’s “Mother of All Business Questions”
And the ministry’s next Best Evidence Synthesis attempts to answer the educational equivalent of what leads to high performance in terms of school leadership practice.
I am restlessly waiting for the release of the full BES report so that I can better critique the claims made about school leadership styles. I want to look at the independence of the data used to generate this report in the ways outlined in Phil Rosenzweig’s The Halo Effect.
Before the full report is released my critique has to rely on comments in an interview with Professor Viviane Robinson, one of the study author’s, to suggest that this Best Evidence Synthesis may well be fundamentally flawed.
Close reading of Professor Viviane Robinson’s comments in the Gazette suggest that this BES is shaped by Rosenzweig's nine delusions that deceive managers - “errors of logic and flawed judgements that distort our understanding of the real reasons for high performing” schools
Delusion 1: The Halo effect
This BES sounds like a classic example of Rosenzweig’s Halo effect thinking where we make inferences about specific traits on the basis of a general impression.
“Perhaps nothing lends itself to the Halo Effect more than leadership. Good leaders are often said to have a handful of important qualities: clear vision, effective communication skills, self-confidence, personal charm and more. Most people would agree these are elements of good leadership. But defining them is a different matter altogether, since several of these qualities, tend to be in the eye of the beholder – which is affected by company performance.” The Halo Effect P58
I suspect this is the case when I read
“Professor Viviane Robinson at Auckland University says emerging research clearly shows that the leadership of schools where students perform above and below expected levels is quite different.”
However, she cautions that the question of whether school leaders do make a difference is another story. Studies which measure undifferentiated samples of school leaders and test the effect they have on students usually show that school leaders have a weak and indirect effect on students.
"In other words, if you take a random sample of schools, it all sort of washes out. But if you compare the high performers and the low performers in a sample of otherwise similar schools, the leadership looks very different. Educational Gazette 86 11 2007
My suspicions that the research data used in the study is flawed with Halos is affirmed by comment that it is only when comparing high performing schools with low performing schools that significant differences in leadership practice emerge.
The Halo Effect occurs when attributions made about leadership practices depend upon the school’s performance. In a school that is perceived as performing well the attributions made will be uniformly positive, in schools deemed to be performing poorly the reverse occurs.
Understanding the Halo Effect allows me to expose a vulnerability in claims in the Gazette article that comparing leadership practice in high and low performing schools is a “particularly helpful”.
"Those studies are particularly helpful when they enable us to identify which particular leadership practices are making the difference to students. Educational Gazette 86 11 2007
As Rosenzweig explains “we have no satisfactory theory of effective leadership that is independent of performance.”
The Gazette article claims that the ministry's next BES reveal leadership dimensions associated with high performance schools.
“From this analysis, Viviane and her team have identified five leadership dimensions that have a particularly powerful impact on students.” Educational Gazette 86 11 2007
1. Establishing goals and expectations:
2. Strategic resourcing:
3. Planning, coordinating and evaluating teaching and the curriculum:
4. Promoting and participating in teacher learning and development:
5. Ensuring an orderly and supportive environment:
Since a wide range of behaviours can be shown to fit the leadership criteria identified by Robinson and her team, it is critical to look closely at how these were determined to ensure the data collected has not been undermined by the Halo effect.
I need to check that the research this synthesis relies upon is unaffected by the performance of schools themselves, so that the independent variables (measures of leadership practice) are measured separately from the thing we are trying to explain (high and low school performance). For example if the data used relies on perceptions in teacher surveys from the schools themselves then we are likely to collect attributions based on performance rather than independent measures.
Delusion 2: The Delusion of Correlation and Causality
The second delusion I want to check for in this BES is “inferring causality from correlation”
I will be looking at the research studies used in the BES for findings based on longitudinal studies.
For example, I need to ask does “promoting and participating in teacher learning and development” lead to a high performing school, or do leaders in high performing schools have more opportunities for “promoting and participating in teacher learning and development”? One way to check for this is to see if any of the research used to support the BES synthesis tracks performance and leadership criteria longitudinally rather than cross sectional data.
Delusion 3: The Delusion of Single Explanations
The third delusion I’m looking for (if we can discount Halo effects and delusions of correlation and causality from the ministry’s Best Evidence Synthesis) is whether leadership practice is a separate effect on school performance or additive.
I want to determine
How much does leadership practice overlap with other influences on the performance of schools?
When a school changes principal how much of the improved school performance is attributable to their leadership practice and how much is attributable to other factors?
I hope I am wrong but I suspect that the BES focus on leadership and high and low performing schools will lead educators to think of school performance in terms of “single explanation” thinking about leadership practice.
Delusion 4: The Delusion of Connecting the Winning Dots
When a Best Evidence Synthesis is made up entirely of research evidence from best performing schools, or even best and poorest performing schools we are in danger of connecting the winning dots delusions. The Halo effect complicates this – unless the data is gathered in a way that is independent of performance there is no explanation of leadership practice against school performance. I really need to check that the research data that the BES uses is collected from performance independent measures across schools at all performance levels.
This is an interesting challenge given the disclaimer by Professor Viviane Robinson that
"Studies which measure undifferentiated samples of school leaders and test the effect they have on students usually show that school leaders have a weak and indirect effect on students.
"In other words, if you take a random sample of schools, it all sort of washes out. But if you compare the high performers and the low performers in a sample of otherwise similar schools, the leadership looks very different."" Educational Gazette 86 11 2007
Delusion 5: The Delusion of Rigorous Research
“We can do our best to select samples of high performers and low performers but if the data are coloured by the Halo effect, we will never know what drives high and low performance; instead we will merely find out how high and low performance are described.” The Halo Effect P 100
“Of course, we don’t see how problematic the research really is, thanks to another delusion : the Delusion of Rigorous Research.”The Halo Effect P 100
The Gazette interview with Professor Viviane Robinson from the Faculty of Education at the University of Auckland with its “ 25 such studies have been located so far,” is drizzled with Rosenzweig’s “We were thorough.” “We were exhaustive.” “We speak with authority”
The BES claims that
Evidence about the links between leadership and student outcomes was provided from 24 studies published between 1985 and 2006.
The majority of studies was conducted in United States schools. Two studies were conducted in Canada and one only in each of Australia, England, Hong Kong, Israel, Netherlands, New Zealand and Singapore.
Fourteen studies examined leadership in elementary school contexts, three in high schools, and seven studies included a mix of elementary, middle and high schools.
Thirteen of the studies confined their analysis of school leadership to the principal only, while 11 took a broader, more distributed view of leadership.
While the studies examined the impact of leadership on a wide range of student outcomes, academic outcomes – notably maths, reading and language skills – predominated.
The four studies that examined leadership impact on students' social and personal well-being included measures of attitudes to school, teachers and learning, and students' academic self-concept, engagement with their schooling, and retention rates. Educational Gazette 86 11 2007
But as Rosenzweig notes
“if your data sources are corrupted by the Halo Effect , it doesn’t matter how much you’ve gathered. You can stack Halo’s all the way to heaven, but you’ve still only got Halos.”The Halo Effect p101
Delusion 6: The Delusion of Lasting Success
The findings of high and low performance school leadership practice in the BES suggest sustainability. But sustaining high performance in schools is hard. Implicit in the BES is the suggestion that one particular collection of leadership practices aligns to school performance that will sustain over time.
I will be interested to look for the research in the BES that tracks performance over time.
Performance is essentially competitive. Rosenzweig’s research suggests that competitive advantage is hard to sustain over time, suggesting that highly performing schools are unlikely to continue as highly performing schools over time.
It seems more honest to acknowledge that “Imitation, competition and expropriation” means that (school) performance over time has a “tendency to move toward the middle a clear regression towards the mean” The Halo Effect p105
Delusion 7: The Delusion of Absolute Performance
Performance is relative. Focusing on leadership practices associated with high performing schools suggests that a school’s high performance is in some way independent of the performance of other schools. But in truth the performance of any school will always be affected by the performance of other competing schools.
As a principal, your leadership practice impact on student achievement and well being is affected by how other schools are performing. Your impact is not absolute.
Suggesting that you can achieve high performance, regardless of what other schools are achieving may well make the BES more palatable but is a delusion.
Delusion 8: The Delusion of the Wrong End of The Stick
How do we interpret our findings?
Do all school leaders who establish: goals and expectations, strategic resourcing, planning, coordinating and evaluating teaching and the curriculum, promote and participate in teacher learning and development and ensure an orderly and supportive environment do better?
When we focus on schools where students perform above and below the expected levels we exclude the middle. We cannot say on average whether principals who adopt the five leadership dimensions will do better overall than principals who don’t.
I need to look at the research data this BES is based upon to determine whether it looks broadly enough at school performance to claim that one vocabulary of leadership practice will enhance the chances of school performance success.
Delusion 9: The Delusion of Organisational Physics
“The emphasis on certainty, on clear causal relations rather than contingency and uncertainty, illuminates one final misconception. ...” The Halo Effect
That we might discover “timeless universal answers that can be applied to any organisation.” in the ministry's BES is delusional.
To paraphrase Rosenzweig
“We can’t put [schools] in petri dishes and run neat experiments. And since even the best studies of [schools], ones that carefully follow stringent research methods , ones that make sure to avoid Halos and that control for rival variables and make sure not to confuse correlation with causality, can never achieve the precision and replicability of physics, then all claims of having isolated immutable laws of organisational performance are unfounded.” The Halo Effect P126
Perhaps "The Halo effect ... and the Eight other Business Delusions that Deceive Managers" is what Professor Viviane Robinson alludes to when she states
“We need to be cautious, however, as there are very few studies which have tested the links between leadership and student outcomes." Educational Gazette 86 11 2007
Or perhaps Red Dwarf’s Lister is “ontoit” and the ministry's next BES on leadership really does tell us how to become a "brilliant general".
well this BES paper should be renamed BS paper. that or my school could do with some decent leadership.
“From this analysis, Viviane and her team have identified five leadership dimensions that have a particularly powerful impact on students.” Educational Gazette 86 11 2007
1. Establishing goals and expectations:
2. Strategic resourcing:
3. Planning, coordinating and evaluating teaching and the curriculum:
4. Promoting and participating in teacher learning and development:
5. Ensuring an orderly and supportive environment:
i am striking out here, yeah sure we have a plan 1. we resource teachers 2. we help teachers get better 3. i sit in on PD when able 4. we shout on a friday 5. but to run a "comprehensive" study with a bunch of crapiola research from 24 schools from the other side of the world, and draw comprehensive conclusions only from the top and bottom of the survey starts to smell like homer after a big night at Moes.
24 studies published between 1985 and 2006.
The majority of studies was conducted in United States schools. Two studies were conducted in Canada and one only in each of Australia, England, Hong Kong, Israel, Netherlands, New Zealand and Singapore.
Fourteen studies examined leadership in elementary school contexts, three in high schools, and seven studies included a mix of elementary, middle and high schools.
Thirteen of the studies confined their analysis of school leadership to the principal only, while 11 took a broader, more distributed view of leadership.
While the studies examined the impact of leadership on a wide range of student outcomes, academic outcomes – notably maths, reading and language skills – predominated.
The four studies that examined leadership impact on students' social and personal well-being included measures of attitudes to school, teachers and learning, and students' academic self-concept, engagement with their schooling, and retention rates.
that paragraph alone shows its a shakey study.
the sad thing for me is that endless principals will sit around in BES learning circles and discuss the outcomes of this BES study. we will be sent pages of crap about this BES reading and the MOE will quote this leadership shit in future funding "initiatives" and expect us to have bought into it all.
I dont think we challenge research enough. i reckon its ok to say bullshit, and be totally uninformed, and the academic cant be argued with but it downright pisses them off when the uninformed disagree without validity.
if proven leadership traits are there, all well then good, but you cant teach personality. some of our leaders just need get one. and the researchers too.
Posted by: Luke | August 20, 2007 at 12:02 PM
I dunno Luke, without the Halo effect this BES on leadership and its explicit links to student outcomes could be groundbreaking stuff
The BES on leadership could be powerful new educational research from New Zealand that leads the world in getting better learning outcomes for kids through the identification of best practice principles for principals
– and I think we’d both agree that improving learning outcomes would be something of a “good thing” – and perhaps even worth spiralling around a few times in one of those BES Learning circles.
What makes me suspect that this not the case is pasted below [Other research affirming this study is cited in the book]
“The Halo Effect explains how people think about decision processes, an organisation’s people, and leadership – and it doesn’t go away when we conduct large scale surveys, either. Quite the contrary. If we’re not careful, surveys might be little more than a large collection of Halos.p61
This means that those 5 leadership dimensions are much more likely to be ATTRIBUTES ascribed to school principals on the basis of their schools perceived performance - descriptors of how people tend to describe the leaders of schools they believe to be “high performing schools” than they are to be dimensions that CONTRIBUTE to high school performance.
That is - the same principal would find their leadership dimensions described differently if the people surveyed believed the school to be a low performing school. Does mean you are probably safe in terms of how your leadership dimensions are perceived Luke – in the short term anyway.
Finding out what people ATTRIBUTE to the management approaches of principals working in high performing schools (a Halo Effect delusion), may well be interesting in an academic kind of way but is not valuable enough to support many quality learning circle discussions amongst principals.
Finding leadership dimensions that CONTRIBUTE to school performance will be valuable enough to support principal learning circle discussion. BUT we can only ever pretend to this if our data can be shown to be independent of performance
Our first challenge when the full report comes out(addressing Delusion #1) requires us to unpack the original research. We need to figure out if this BES is simply a research synthesis of the descriptions of the five school leadership dimensions that people commonly attribute to high school performance, or a research synthesis of performance independent studies that identify the dimensions of school leadership that contribute to high performance.
The critical question seems to be “Is the data that supports the 5 dimensions gathered in a way that is independent of performance?”
If it is it will be something of a world first in terms of measures of effective leadership.
Then we can critique the report against the other 8 delusions
Posted by: Artichoke | August 20, 2007 at 05:43 PM