“A test is valid only if it measures that which has actually been taught.”
How many of you agree with the above statement? Offhand, it sounds quite reasonable, doesn’t it? But I would say that it holds true only for a minority of tests, specifically achievement tests, and doesn’t make a lot of sense if applied to placement, diagnostic, aptitude, or many proficiency tests. For example, if you are testing a candidate for a position in your new Kickstart venture, you should be asking questions that measure the person’s suitability to the future job position, not how well they absorbed their high school lessons.
I was in Kagoshima (in southern Japan) in September to attend the national JACET (Japan Association of College English Teachers) convention, and, piqued by a symposium offering new perspectives on creating and developing university entrance exams – always a research and practical interest of mine, decided to attend.
The quote at the top of this post was the first item on the presentation slides. I thought it would be challenged. It wasn’t.
The four symposium presenters did offer an array of thoughtful, pedagogically sound, and linguistically progressive views as to how university entrance exams, could engage a more holistic and communicative view of English. Focuses upon productive skills, critical thinking, strategic competence, prediction and summarizing skills were among those advocated, with numerous task samples offered. I even had the pleasure of seeing one of my published papers on this topic quoted in one of the presenter’s slides.
Now, I don’t think anyone reading this blog would deny that testing a wider variety of skills, particularly skills that display higher-order thinking and comprehension, would be a wonderful and accurate measure of a person’s actual English ability.
So, In the follow-up Q&A I told the presenters that I agreed with their assessments as to what types of evaluation should be placed on university entrance exams, and that in fact some universities (cough – mine – cough) have been doing just that.
But, I added, did they really believe that in order to be valid a test must measure that which has been taught? I asked this without malice (I never want to put a speaker on the spot in Q&A) and they said, yes, they thought it was a good maxim. Now, you see where this is going, don’t you? If they think that a valid university entrance exam must be based upon that which has been learned in high school, then the fine ideals that they proposed for the university entrance exams would all be moot, because – as every schoolboy knows – these skills are generally not addressed in Japanese high school English classes.
In other words (and I tried to make this argument at the time), if we insist upon employing this maxim, then university entrance exams should focus mainly upon sentence diagramming and receptive minutiae because these are what is generally practiced in high schools (usually due to the belief that this is what university entrance exams demand). So, maintaining the maxim simply leads to the vicious cycle of high school teachers saying they have to teach this way because the university entrance exams demand it, while the university entrance exam makers (apparently) feel they must only test those things practiced in high school. The result? Static. No one budges.
I argued, and still argue, that university entrance exams should be considered placement and proficiency exams. After all, they are primarily forward looking. You are, presumably, trying to decide which candidates are best suited to participate in the particular faculty/program/school you are operating. You should not attempt to do this primarily by measuring who got the best English results in high school.
But the lead presenter of the symposium, actually the head of the testing special interest group in JACET, insisted that entrance exams are not placement tests, but rather achievement and proficiency exams. I wanted to reiterate my point about how treating it as an achievement exam basically pulls the theoretical and pedagogical good practice carpet from under your feet, since you will then be bound to the “must test what was taught to be valid” maxim, and all your good testing intentions go up in smoke. But I didn’t want to hog the Q&A session…
It may seem like a minor item but as long as we think of, and design, university entrance exams as achievement tests, we are held hostage to often questionable high school classroom methods and practices. If we treat them as future-oriented placement tests we can address more holistic English skills, with a subsequent washback effect throughout the system.
What do you guys think?