Site Search  
RSS feed

Teacher newsmagazine

BCTF Online Museum
BCTF Advantage
Teacher Newsmagazine Volume 20, Number 3, November/December 2007

Confessions of two FSA markers

Much of the attention on Foundation Skills Assessments (FSAs) has been on the enormous waste of time involved, the stress on students who are unaccustomed to this kind of testing, and the abuse of the tests by organizations like the Fraser Institute who use the numbers to "rank" schools. But as if these were not reasons enough to call for the end to these tests, it turns out that the marking of them is just as questionable as the tests themselves. Here are two stories—confessions of FSA markers—or rather, as one points out, coders.

Speed was a must

I have kept this story to myself for quite a while, but having finally given up my dream of being appointed a superintendent of achievement, I suppose it can’t do much harm to tell it.

In the summer of 2001, I agreed to be a marker for the Grade 7 Writing component. After our training, it was clear that speed was an absolute must. As the white boxes were wheeled in, it was explained that, on no account was there any budget to work longer than the allotted five days. We had to get finished by Friday. We were also promised we could leave early on the last day if we were finished. We were constantly told to go faster; indeed we were specifically told NOT to read the whole essay! This was told to us not once but several times. "Read the first part, get a sense of the quality, and put down a number from one to five." As for going back to reconsider a mark, or pausing to confer—not a chance. The sheets whizzed by in a blur. By the way, they don’t even call it "marking." Apparently, the term for what we were doing is "coding."

But even more surreal was my later role as a member of the Provincial Standards Setting Committee. We were directly supervised by two ministry staff and it was our job to decide where to place the cut-off grade for Meeting Expectations. The process (I’m totally serious here) was that we FIRST decided what percentage of the province should meet expectations, THEN we looked at the raw scores and calculated what threshold would make it "come out right." Well, we were getting paid, so... I can’t remember the exact numbers now, and as usual we had to leave every scrap of paper behind, but I swear it’s the truth.

– Patrick Truelove is a Delta teacher.

Behind the scenes in an FSA marking mill

Do FSA scores provide a reliable measure of the abilities of students in BC schools? My experience as an FSA marker does not inspire confidence. In 2002, needing the money, I volunteered to mark Grade 7 English FSA essays. A room full of teachers worked for five days at SFU under the supervision of facilitators from the ministry.

The first day’s orientation included about an hour devoted to explaining the marking criteria. The purpose of our marking was explained to us, and we were given a rubric for judging whether the essays would be classified as 1 (the lowest), 2, 3, or 4. This was followed by a practice session. Markers would work in pairs with a bundle of 20 or 30 papers at a time.

Once we started, the tone gradually changed. We soon realized that the top priority was speed. The marking used to be done in seven days, so the reduction of time to five days greatly increased the pressure to work quickly. Each daythe facilitators told us our group was doing fine but that we still needed to speed up.

On the third day, they started writing on a chalk board the number of boxes of essays that had been completed and how many still had to be done. The supervisors also began announcing the range between the pair that had completed the largest number of bundles and the pair that had done the least. People who complained that there was no way to mark faster and still read the whole essay were humiliated by the rest of the group. During the week, at the urging of the facilitators, the overall pace constantly increased. The fastest markers were openly praised for their speed and accuracy. The facilitators paired faster markers with slower ones in hopes of speeding them up. Fast markers were told they would be hired back again.

Eventually it became clear that markers were not reading the entire essay. One of the faster markers admitted that he only looked at the last paragraph and only skimmed it at that, spending no more than 10 seconds per booklet to assess, mark, and initial. At no time did the facilitators ever tell markers not to read the whole composition, but they did have the speedy ones demonstrate and even give verbal directions on how to mark quickly by skipping over almost all of each essay.

Some discovered an even faster method. Given that 95% of the papers were arguably 2s, one could achieve 95% accuracy just by marking every paper with a 2.

I marked bundle after bundle with 2s and was told my work was accurate.

The only apparent accountability was the inclusion of a reliability paper in some of the bundles. However, this paper was not only obviously photocopied but was actually labelled "Reliability Paper." Of course, the partners examined and discussed this type of paper meticulously and made sure each of them recorded the same mark for it.

On the last day, our group was rewarded for its speed by being dismissed early after a 90-minute discussion session about FSAs.

– Anonymous

Source: FSA Examiner, Special Edition, April 24, 2003, Surrey Teachers’ Association.

  • FacebookTwitterYouTube
  • TeachBC
  • BCTF Online Museum
  • BCTF Advantage