“Anon comes good” Yeah right!

I
I teach
I teach kids
I teach kids how
I teach kids how to
I teach kids how to evaluate
I teach kids how to evaluate gen(ral).eyes.(aye) shons
I teach kids how to evaluate
I teach kids how to
I teach kids how
I teach kids
I teach
I

... well I guess I struggle to comprehend why no one bothers to teach the same stuff about generalisation to adults

When I read Leigh’s Teach and Learn Online Blog post –  “Anon comes good”  I wished, I wished, I wished grown ups knew how to evaluate generalisations

When I teach kids the steps to making a “good” “generalisation” I get them to look at the generalisation

I ask them to describe the sample used to support the generalisation (reliability), and then I ask them to judge the sample selection (validity) as adequate or inadequate to support the generalisation,

I ask them to do this  before they decide whether to accept, to reject, or straddle the fence with both buttocks,  (or even with one buttock – to straddle is to straddle after all)

I ask them to do this before they accept the generalisation as a good (reliable and valid) generalisation

Leigh, seems prematurely, precipitately, rashly, impulsively  captured, ensnared, even tied up in the temple [highlight most appropriate descriptor]  by the generalisations in Anonymous’s link to The ICT in Schools Report 2005 prepared for the 2020 Communications Trust; - tied up to generalisations like

The “Three quarters of all schools”“ Generalisation

Schools are demonstrating a strong interest in e-learning, with the technology infrastructure largely already in place. Moreover, around three quarters of all schools are indicating plans for implementing at least one e-learning activity during the next 12 months, or even the

The “Demonstrating Leadership” Generalisation

In some specific areas such as linking with other schools by video or audio conferencing for virtual events, Māori Medium schools are demonstrating leadership.”

I blame the dreary weather in Dunedin and possibly the courtship behaviour of the albatross –it has obviously blunted Leigh’s usually acute critical acumen

When you pause to look for the sample used (sample reliability) to support the "three quarters of all schools" generalisation, or the "demonstrating leadership" generalisation - and you judge the sample selection (validity) you discover that

The ICT in Schools Report 2005 prepared for the 2020 Communications Trust reports on the self report responses from 285 principal surveys and 272 equipment questionnaires – questionnaires were sent out to to 800 schools. [you can work out the response rate reliability measure for yourself] – Impressive eh!

A response rate of 285 out of 800, is also a non-response rate of 515 out of 800.  Suggesting that 64% of New Zealand school principals surveyed about professional development, ICT planning, management and use in the school, spending on ICT and attitudes towards ICT did not bother to complete the survey.

As to the validity of the “what is actually happening stuff” I leave the fact that the analysis is based on data collected from a “self response” ( a “telling the data collector what we think they want to hear exercise ) survey of school principals’ to your critical imagination.

So how valid are the generalisation statistics?   

The ICT 2005 report writers talk about their own data wrt

The three quarters of all schools generalisation:

“It is also important to note that survey results may show slightly more ICT use in schools than is actually the case. The bias in the results may occur if schools more active in using ICT disproportionately tended to respond to the survey. All results may be subject to non response bias which is an unknown that cannot be calculated. Caution should therefore be used when interpreting these survey results.

The demonstrating leadership generalisation:

Or how the responses from 28% or twenty seven schools can be represented as a “demonstrating leadership” generalisation.

“Included in the sample of 800 schools were all 96 Maori Medium schools, of which 27 Principals’ and 27 equipment questionnaires were returned.  Of the 27 returns , 17 were from Te Kura Kaupapa Maori schools, 8 from bilingual schools , and 2 from Secondary Maori boarding schools.  Due to the small sample size, these school types have been combined into Maori Medium schools.  Therefore the response rate for Maori Medium schools is 28% for both questionnaires.

If you were moved by the generalisations in the ICT in Schools Report 2005 I think you will be gob smacked by an overwhelming swash of déjà vu when you check out Christopher D Sessums blog post on what appears to be a very similar initiative -  Polling, scientific research, and sources of error So similar in fact I hope we didn't pay too much for the design of the questionnaire we used.

Sessums asks the harder questions that needed to be asked of this ICT in Schools Report 2005 before the self response data from 285 schools is used to draw generalisations about ICT use in all New Zealand schools

The article is based on a report sponsored by CDW-G, a reseller of hardware tools to schools and governments, and administered by education research firm Quality Education Data (QED). The study, called Teachers Talk Tech 2006: Fulfilling Technology's Promise of Improved Student Performance, polled “some 1,000 K-12 public school teachers on technology's role in the classroom.”

I was both impressed and depressed by the above findings: Impressed with the number of teacher who reportedly see the value of properly integrating technology into their classrooms and who feel comfortable using technology, and depressed by the lack of scientific rigor associated with the results.

Sessums elaborates on issues of sampling error, selection bias, response bias, wording of questions and coverage bias in self reporting questionnaires.

As an educational researcher, I am skeptical of the results in that the poll’s methodology is not clear about which teachers were chosen for these results, how they were chosen, the questions that were asked, the validity and reliability of the poll, etc.

We may want to pretend that Anonymous has come good, but Sessums analysis, and my predilection for howling at the moon makes me wish that both grown ups and kids could learn how to evaluate generalisations.