Artichoke

If only Jane Gilbert had a blog .... and NZCER had a wiki ....

Last week a local principal gave me a copy of Jane Gilbert’s  “Progress” in 21st century education? – a New Zealand Council for Educational Research conference paper.   In it Gilbert claims that we need to rethink “progress” and think differently about what it is to learn.  Apparently schools need to change to better meet the needs of 21st Century Learners and Gilbert postulates this might happen if we;

• build learning capacity,
• develop deeper and richer networks and links for core competencies,
• think together and develop collaborative teams of learners, and
• find ways to ensure that everyone achieves.

It feels kind of strange to be learning about the need to change the way we learn in the 21st Century  in a 21st Century conference paper that is accessible only to those prepared to attend the conference or to pay for the print copy of the conference proceedings. 

The experience of reading the conference proceedings was in truth a retreat to a past way of learning.  The freedom that comes from living outside of the institution in the day job is balanced by relying heavily on accessing my professional learning online rather than from print sources or from officially sanctioned professional reading circles. My ability to interact with content and ideas in Gilbert's paper seemed limited in some way ... In reading the NZCER conference paper I felt like I was practising pedagogical austerity.  

So it was ironical to read future focused calls for schools to change “the way they do learning” when the researcher and research organisation calling for change is operating in an environment where;

• content is still centralised,
• communication remains in the form of a monologue with no opportunity provided for online collaborative comment, trackback  or conversation,
• learning is all about push over pull, and
• where unlike a wiki document, the learning manuscript is presented in its final form, ensuring that the learning process and editing involved in creating this thinking is not available for scrutiny.    

It is a little like the teacher who claims to value discussion and student questioning for learning when talking to other educators but whose educative classroom practice reveals that for their students “learning is listening”.

Our limited access to Jane Gilbert’s paper, “Progress” in 21st Century Education? is worth thinking about. 

It is worth challenging why we are so ready to accept espoused values of educational researchers in New Zealand when the integrated or lived values of their institutions are so different.  If only Jane Gilbert had a blog.... and NZCER had a wiki.... how different would our learning opportunities be?

My experience working with and in schools in the day job suggests that the ideas in “Progress” in 21st century education?” are dated.  Especially the ideas around how to measure the complexity of what a person is able to do in real- world situations.  After all Eisner was talking about curriculum connoisseurship over 20 years ago.  But I have no easy way of testing these ideas in the context of the paper, and I guess there is often a dislocation between what educational researchers claim is happening in schools and what schools are actually doing...

Many of the educators we work with in the day job would claim that Gilbert’s building learning capacity, developing deeper and richer networks and links for core competencies, thinking together and developing collaborative teams of learners, and finding ways to ensure that everyone achieves, are already with us.   And unlike the NZCER conference paper, New Zealand schools are no longer broadcasting their learning outcomes solely through print copies of the school newsletter or at meet the teacher evenings where the teacher shares a paper sample of a student learning outcome with the parent.  

A virtual (or face to face) tour of students’, teachers’ and principals’ practice would reveal that primary, intermediate and secondary schools across New Zealand are already building learning capacity, rich networks, collaborative teams, learning through participation and collaboration.  And they are using social media with their students, their teachers and their communities  to help them do this - including using wikis, blogs, podcasts, vodcasts, social bookmarking, tagging, online photo libraries, Diigo, Google docs, Open notebook, shared online calendars, aggregators, RSS feeds etc.

In fact there are so many of these 21st Century appropriate learning conversations going on that I sometimes worry about the adverse influence of social media on the relevance of the local.

I increasingly want to explore Mejias’ question 

Does social media contribute to the irrelevancy of the local?

In the words of Lowell Monke: As the computer has amplified our youths' ability to virtually "go anywhere, at any time," it has eroded their sense of belonging anywhere, at any time, to anybody, or for any reason.   Monke, 2005, Charlotte's Webpage: Why children shouldn't have the world at their fingertips

I want to ask

Does social media create learning networks that discriminate against the student's immediate surroundings?

Or in the context of the Auckland ICTPD Cluster Home Group Meetings

Does social media create learning networks that discriminate against an Auckland ICTPD facilitator’s immediate surroundings?

It doesn’t seem unreasonable to claim that in embracing social media ... in creating our own personal learning networks (all that Twitter, Skyping, blog reading and commenting, wikis, YouTube and TED talks and or podcasts vodcasts etc) we have increased the irrelevancy of the opportunity to meet f2f to learn with/ from our immediate companions in our local surroundings in a Home Group Meeting.

After all if you consider that the learning networks available to those attending the Auckland Home Group Meeting ... they rely largely upon happenstance ... who decides to turn up, for how long on the day .... It seems we cannot even count upon the National team members attending given that “This is not a day where the national team deliver content - this is a day for you all to participate and contribute.”

There is an interesting tension here.... in advocating for the increased use of social media for building learning networks by educators we undermine the relevance of the local ... the opportunity to attend the Auckalnd Home Group Meeting for building learning networks. 

This argument holds even when looking at the “loss leader” ... the “outside expertise” attendance enticer.... it seems plausible that the opportunity to learn from/network  with  the “outside expertise involved in the session” would lose some of its relevance if this same presentation and shared expertise were available online through social media .

All this thinking makes me wonder  

How should we re-approach nearness at the Auckland Home Group Meetings?

And this in turn makes me suspect that “Progress” in 21st Century education?  will end up being an exploration of: 

How should we re-approach nearness in our schools?

References:
“Progress” in 21st Century education? By Jane Gilbert Paper presented at the New Zealand Council for Educational Research (NZCER)  Conference: Making progress - measuring progress, Wellington, 13 March 2008. p.63-73
URL: http://www.nzcer.org.nz/default.php?cpath=139_133&products_id=2194 

Re–approaching nearness: Online communication and its place in praxis by Ulises A. Mejías
First Monday, volume 10, number 3 (March 2005),
URL: http://firstmonday.org/issues/issue10_3/mejias/index.html

Dolly Parton and the MOE’s Best Evidence Research Synthesis

Given that so much of what “we do” in education in New Zealand is directed by meta-analysis, including all our Best Evidence Research Synthesis reports, reading Gene Glass on “Meta analysis at 25”  makes me feel insecure and vulnerable ...

For .... much like reading Phil Rosenzweig’s The Halo Effect  undermined my faith in the Best Evidence Synthesis on School Leadership and Student Outcomes.   .... Glass’ meta-analysis critique promises to betray me ...

It is important to note here that I don’t know enough statistics to refute Glass.  This lack of statistical referencing probably explains why, as I read Glass’ paper, I am reduced to imagining myself Dolly Parton-ised, belting out “Jolene, Jolene, Jolene, Jolene ...I’m begging of you please don’t take my man  ...” 

If I sing loudly enough perhaps I will block the following Jolene thoughts from undermining all the MoE “evidence based practice” “Best Evidence Synthesis” stuff in my mind

Jolene Thought #1: Does the process of selecting and discarding studies as described in the School Leadership and Student Learning Outcomes BES  equate to Glass’ censorship by some a priori set of prejudices?

.... newly minted confections such as "best evidence research synthesis," a come-lately contribution that added nothing whatsoever to what myself and many others had been saying repeatedly on the question of whether meta- analyses should use all studies or only "good" studies.

I remain staunchly committed to the idea that meta-analyses must deal with all studies, good bad and indifferent, and that their results are only properly understood in the context of each other, not after having been censored by some a priori set of prejudices. An effect size of 1.50 for 20 studies employing randomized groups has a whole different meaning when 50 studies using matching show an average effect of 1.40 than if 50 matched groups studies show an effect of -.50, for example. Glass 2000 Meta-Analysis at 25

Jolene Thought #2: Is it valid to make inferences from meta-analysis generated “effect sizes” when the nature of the studies used means that meta-analysis must lack both “a well defined population that has been randomly sampled” and “subjects that have been randomly assigned to conditions in a controlled experiment”?

Moreover, the typical meta-analysis virtually never meets the condition of probabilistic sampling of a population

It is common to acknowledge, in meta-analysis and elsewhere, that many data sets fail to meet probabilistic sampling conditions, and then to argue that one ought to treat the data in hand "as if" it were a random sample of some hypothetical population. One must be wary here of the slide from "hypothesis about a population" into "a hypothetical population." They are quite different things, the former being standard and unobjectionable, the latter being a figment with which we hardly know how to deal.

...... If the sample is fixed and the population is allowed to be hypothetical, then surely the data analyst will imagine a population that resembles the sample of data. If I show you a handful of red and green M&Ms, you will naturally assume that I have just drawn my hand out of a bowl of mostly red and green M&Ms, not red and green and brown and yellow ones. Hence, all of these "hypothetical populations" will be merely reflections of the samples in hand and there will be no need for inferential statistics. Or put another way, if the population of inference is not defined by considerations separate from the characterization of the sample, then the population is merely a large version of the sample. With what confidence is one able to generalize the character of this sample to a population that looks like a big version of the sample? Well, with a great deal of confidence, obviously. But then, the population is nothing but the sample writ large and we really know nothing more than what the sample tells us in spite of the fact that we have attached misleadingly precise probability numbers to the result. Glass 2000 Meta-Analysis at 25

Jolene Thought #3:  Are the statistical tests used to determine homogeneity in meta-analysis studies adequate for the task?

Once a hypothesis of homogeneity is accepted by Hedges's test, one is advised to treat all studies within the ensemble as the same. Experienced data analysts know, however, that there is typically a good deal of meaningful covariation between study characteristics and study findings even within ensembles where Hedges's test can not reject the homogeneity hypothesis. ..... The best data exploration and discovery are more complex and convincing than the most exact inferential test. Glass 2000 Meta-Analysis at 25

However, my biggest Jolene Thought Moment came on reading “Another reason why I'm leery of meta-analyses”  on the Respectful Insolence Blog (thanks to link from Stephen Downes)

Jolene Thought #4:  Are meta-analyses nothing more than systematic reviews of the literature with attitude?

The Respectful Insolence Blog post describes an investigation into the use of meta-analysis in evidence based medicine (EBM),  which does not sound all that dissimilar to the use of BES in New Zealand education.

One increasingly common method of trying to make sense of the morass of data addressing various clinical questions is the medical literature phenomenon known as meta-analysis. A meta-analysis is different from a clinical trial in that it is a statistical reanalysis of data from existing trials. Generally the highest quality trials are chosen in accord with the principles of EBM, and the data from these trials is all lumped together and analyzed in order to provide in essence a more rigorous treatment of existing data than a systematic review of the literature. To be subject to meta-analysis, a medical question must have multiple studies addressing it, and the results must be quantitative. Orac The Respectful Insolence Blog

The post goes on to describe an investigation into the meta-analysis of “randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period.”

The investigators found that 

“....given the same studies and the same extracted and abstracted data, different investigators came to very different conclusions. This is in contrast to the dogma that tells us that meta-analyses represent the most objective method of reviewing large bodies of studies. This study casts considerable doubt on this contention, as the authors point out:

Although systematic reviews with meta-analyses are considered more objective than other types of reviews, our results suggest that the interpretation of the data remains a highly subjective process even among reviewers with extensive experience conducting meta-analyses. The implications are important. The evidence-based movement has proposed that a systematic review with a meta-analysis of RCTs on a topic provides the strongest evidence of support and that widespread adoption of its results should lead to improved patient care. However, our results suggest that the interpretation of a meta-analysis (and therefore recommendations) are subjective and therefore depend on who conducts or interprets the meta-analysis.

The significance of this study is that it doesn't look at differences in the selection of studies for the meta-analysis or the interpretation of or extraction of data from the studies included in the meta-analysis. Every reviewer was given the same package, the same data, and the same statistical analyses of the included studies, thus eliminating this issue. Even given that, reviewers still interpreted the results of the meta-analyses very differently.

The results of this clever exercise provide just one more bit of evidence that leads me to believe that meta-analyses are nothing more than systematic reviews of the literature with attitude. That's not to say that meta-analyses of the literature aren't often useful, just as systematic reviews of the literature, are. They are in the same way that systematic reviews are: They boil down a large number of studies and suggest an interpretation. Let's just not pretend that meta-analyses are so much more objective than a systematic review as to be considered anything more. Orac The Respectful Insolence Blog

Gene Glass has a future focused suggestion in his paper – one that in 2008 we would most certainly claim as Web2.0 or even Web3.0 thinking ..... he suggests that researchers contribute to online data archives instead of publishing meta-analyses and or BES studies ... it sounds worthy ... it sounds like something that just might let me un Dolly myself and stop singing Jolene ...

Meta-analysis needs to be replaced by archives of raw data that permit the construction of complex data landscapes that depict the relationships among independent, dependent and mediating variables.

Five years ago, this vision of how research should be reported and shared seemed hopelessly quixotic. Now it seems easily attainable. The difference is the I-word: the Internet. In 1993, spurred by the ludicrously high costs and glacial turn-around times of traditional scholarly journals, I created an internet-based peer-reviewed journal on education policy analysis (http://epaa.asu.edu). This journal, named Education Policy Analysis Archives, is now in its seventh year of publication, has published 150 articles, is accessed daily without cost by nearly 1,000 persons (the other three paper journals in this field have average total subscription bases of fewer than 1,000 persons), and has an average "lag" from submission to publication of about three weeks.

Two years ago, we adopted the policy that any one publishing a quantitative study in the journal would have to agree to archive all the raw data at the journal website so that the data could be downloaded by any reader. Our authors have done so with enthusiasm. I think that you can see how this capability puts an entirely new face on the problem of how we integrate research findings: no more inaccurate conversions of inferential test statistics into something worth knowing like an effect size or a correlation coefficient or an odds ratio; no more speculating about distribution shapes; no more frustration at not knowing what violence has been committed when linear coefficients mask curvilinear relationships. Now we simply download each others' data, and the synthesis prize goes to the person who best assembles the pieces of the jigsaw puzzle into a coherent picture of how the variables relate to each other. Glass 2000