Essay Robo-Graders Reward Form, Not Content

If you were appalled by the recently released study results that computers can score test essays as well as humans, take heart.  Machines are fast, but they’re superficial.

As NY Times reporter Michael Winrip notes in “Facing a Robo-Grader?  Just Keep Obfuscating Mellifluously,” ETS’s e-Rater can grade 16,000 essays in 20 seconds.  But as an MIT’s director of writing discovered, the computer falls for cheap tricks–big words, long sentences, empty transitions, and outright lies.

Instructors of writing still have to face the stack, but at least they have some job security.

Advertisements

Update on “The 7 Deadly Sins of Student Writers”

According to Ben Yagoda, punctuation is this generation’s downfall.  See his article in the Chronicle of Higher Ed, “The Elements of Clunk.”

 

Take an Editing Course from the Masters

From the birthplace of Chicago style….

The University of Chicago Graham School of Continuing Liberal and Professional Studies offers individual courses and a certificate program in editing.  Many of the courses meet for a two- or three-day seminar followed by take-home assignments.  Courses include various levels of manuscript editing (often taught by journal editors) and electives like “Editing Electronically: Using Word Processing Tools.”

LRC, Smarthinking, Students in Class: What Can Peer Review Do for You?

In this blog, I give a pep talk for peer review, introduce USF’s CLAQWA rubric, and identify the fall 2011 LRC writing tutors.

As anyone who’s been through a WI workshop knows, I’m a fan of carefully guided peer review.  Even expert writers need feedback from readers as they draft and revise.  Discussing work in progress can engage students more deeply with the material; thinking about how a text conveys its message can building reading as well as writing skills.  Well done, peer review is a win-win assignment—creating an opportunity for student learning and smoothing papers’ roughest edges before you comment or grade.

But many of us have also had experiences when peer review failed, giving students false confidence, for instance, after a round of vacuous “good job” comments.

USF’S RUBRIC

The key to worthwhile peer review lies in the guidance you provide.  When the University of South Florida academic assessment team piloted a structured peer review process in several courses, it made many faculty converts.  Engineering professor Ralph Fehr commented,

“When introduced to the peer review process, I was somewhat skeptical as to how much, if any, it would improve the quality of the students’ writing. I was doubtful that the results would justify the effort put into the review process. After several semesters of participation in the Peer Review Pilot Project, I have seen substantial improvement in the organization and structure of most of the essays evaluated. I have also observed an increased level of collaboration and interaction among the students in non-writing assignments, which I am confident is a result of the peer review training. I will continue to integrate the peer review process into my courses to allow my students to continue to improve their communication skills.”

USF uses a rubric called the CLAQWA.  You can get a sense of it from USF’s 2006 “Assessment Brief.”  I’m on the trail of the revised version mentioned (emails are bouncing), to post on our MUIR wiki.  You can preview it in Appendix B of Irene Suzanne Penner’s 2010 dissertation, “Comparison of Effects of Cognitive Level and Quality Writing Assessment (CLAQWA) Rubric on Freshman College Student Writing.” 

In straightforward language, the CLAQWA looks at 15 “traits,” elaborating five criteria under each one.  I leave one as an example in my outline summary below:

ASSIGNMENT PARAMETERS

Level Trait 1: Assignment Requirements

Level Trait 2: Main Idea

5 The writer clearly has and maintains a main idea throughout.

4 The main idea is clear, although a rare extraneous element is introduced.

3 The paper has a main idea, but additional unrelated ideas distract the reader.

2 The main idea is not maintained or it is unclear.

1 The paper lacks a main idea or appears to reflect the writer’s “free association.”

Level Trait 3: Audience

Level Trait 4: Purpose

ORGANIZATION AND DEVELOPMENT: STRUCTURAL INTEGRITY

Level Trait 5: Opening

Level Trait 6: Coherence Devices

Level Trait 7: Paragraph Construction

Level Trait 8: Closing

ORGANIZATION AND DEVELOPMENT: REASONING & CONSISTENCY

Level Trait 9: Reasoning

Level Trait 10: Quality of Details

Level Trait 11: Quantity of Details

LANGUAGE: CONTEXTUAL AND AUDIENCE APPROPRIATENESS

Level Trait 12: Word Choice

Level Trait 13: Comprehensibility

Level Trait 14: Sentence Construction

Level Trait 15: Point of View

GRAMMAR AND MECHANICS: OBSERVATION OF STANDARD EDITED ENGLISH

Level Trait 16: Grammar and Mechanics [I don’t know why this is out of order in Penner’s appendix]

A number of peer review rubrics/guidelines are circulating on the Web, up for “citation” and adaptation.  I’m happy to go searching for a model that might work for your class.

LEARNING RESOURCE CENTER

Don’t forget the notion of “guidance” if you recommend that your students use the LRC.  You can arm your students with specific questions so that the peer tutor doesn’t have to try to make sense of the whole paper in half an hour.  Your student might say, “My professor asked me to come here to talk about my introduction … to see if my body paragraphs tie back to my thesis … to think about introducing quotes in my paper….”  Remind students to bring your written assignment with them to the LRC.

The LRC will be missing Liz Messman this fall (and moving to a trailer, temporarily, at some date TBA), but it has a great staff.  I hope to have some ongoing conversations with the writing tutors.

I met some of them at the segment of training Liz gave last week:

    1. (veteran) Erica Prong, graduate assistant in the humanities
    2. (veteran—but she’s on study abroad this fall, I believe) Erin Evans, psychology
    3. (new) Krysti Hartman, fashion merchandising
    4. (new) Anna Macedonia, art education
    5. (new) Niven McCall, history
    6. (new) Adrianne Morris, English
    7. (new) Nicholas Papadakis, politics (and philosophy, I believe he said)
    8. (new) Melany Su, biology
    9. (new) Casey Trottier, business

Also on the writing list but not at that training:

  1. (new) Rochelle Coates, graphic design (will do makeup training)
  2. (veteran) Rachael Raske, psychology (she’s done writing training before)
  3. (veteran) Cyndi Trang, biology (she’s done writing training before)

Click here for a list of the entire staff.

SMARTHINKING

There’s now a tab in Blackboard that will take students to Smarthinking (no need to go through tools).  Preliminary data from last year’s pilot suggests a good result: LRC use was up slightly despite the introduction of Smarthinking’s online “e-structor” feedback, which student writers often sought when the LRC was closed.  Again, remind your students to cut and paste from your assignment (this is not the place for guesstimating!) into Smarthinking.  You may ask students to attach a copy of their feedback.

If you’re working with peer review, let me know how it goes.  And please, feed our bank of materials on the wiki!

~Sylvia, x6478

Teaching Tip: Looking back to move forward

At this point in the semester, I’m certainly not going to suggest that you try some new approach to your teaching.   Instead, I’d like to share an idea that could help you develop your own teaching tips.

On my faculty developers’ listserv, we have been discussing the practice of writing an end of semester “case study” of one or more classes.   At one institution, every instructor is asked to critically examine each of their courses and reflect in writing on what went well, what did not go so well, potential ways to improve the course for the next offering, etc. This exercise was not part of the annual summative evaluation (although it might come in very handy), but was meant to provide a useful structure for analyzing and reviewing teaching progress.

Now, I think most of us do some kind of basic looking back at the end of the semester, but I don’t know if anyone at Marymount is doing anything quite this comprehensive (if you are, let me know!).  When I read about it, I first thought that it would take a non-trivial amount of time, particularly if taken seriously.  However, I think that the process would provide benefits that might just make that time worthwhile.  These are the benefits I see (so far anyway):

  • If you only teach a course once a year (or even less), it’s hard to remember what you intended to change unless you keep some kind of records.  While I tend to scrawl a few notes on the old syllabus and assignment sheets and throw them into the course folder, after a year these often seem incomplete and sometimes incomprehensible.  How the future me would appreciate a thoughtful analysis in complete sentences! And since the future me is the only person who gets this, I don’t have to worry about editing it for public viewing.
  • I’m pretty convinced that I would actually process more deeply and learn more from writing up my reflections than even just thinking deeply.  If there is one thing I believe in more every semester, it’s the power of writing to actually help thinking happen, not just record the thinking that occurred.  Writing is learning, and I want to learn things that will help me teach more effectively.
  • I think using this process could help me remember points I want to make about my teaching performance during the annual assessment process, for example by identifying teaching strengths and weaknesses and show how I am addressing them.  So, this process could save time while writing that document, since essentially I will have done some of the work for it in advance.
  • Finally, this process will help me identify where I want to improve as an instructor, what kind of reading I should be doing or what kind of sessions to attend at teaching conferences and what questions I want to ask my colleagues about how they handle specific teaching issues.

A couple of listserv responders suggested possible questions and formats for a course review or case study.  One is fairly structured toward the course itself and asks questions like:

  • What do I think of the course’s learning objectives?  How might they need to be changed and how hard would that be?  Do they feed into other courses or program objectives?
  • How well did students meet each objective?  What evidence am I using and is there better evidence that I could collect?  How well can I even judge how well students met each objective?  Was there an objective I really didn’t measure that well, or at all?
  • Did my assignments help students to meet my course objectives?  Do some of my assignments not really relate to any of my course objectives and goals?
  • Was the time I and my students spent on assignments and activities (e.g. completing assignments, giving feedback for and grading assignments, planning class activities) worth the academic payoff?

Another, more global approach was suggested by independent consultant Alice Cassidy and focuses on the classic paper, Seven Principles for Good Practice in Undergraduate Education, Chickering & Gamson (1987).  Don’t be put off by the title if you teach graduate students.  The concepts are universal.

Good practice in undergraduate education:

  1. Encourages contact between students and faculty
  2. Develops reciprocity and cooperation among students
  3. Encourages active learning,
  4. Gives prompt feedback,
  5. Emphasizes time on task,
  6. Communicates high expectations, and
  7. Respects diverse talents and ways of learning.

Cassidy asks instructors to reflect on how they are implementing each of the 7 principles as well as thinking about the following three questions:

* What do you do in class time, in meetings with students and through design of assignments?  How do you take part in professional development activities to explore more about these?

* In what ways do you document your work through a teaching portfolio/dossier or other material?

* When and how do you explain these to your students?

I’m definitely going to try this approach after my final grades are in.  If you already do something like this, please share your process with us, or other thoughts you have about ways we can learn from our teaching experiences.

 

…and April Is Especially Cruel for Those Who Assign Papers

Those of you who’ve taken the WI workshop series have heard our spiel about how to save time when dealing with papers.

Although it’s likely too late to redesign assignments, you might

  • Consider separating commenting from grading.  Research shows that students usually don’t carry writing feedback from the end of one class into the next semester.  Ever write comments on a paper due the last day of class–and find that many of your students don’t pick them up?  Sigh.

If you give students feedback on a paper in progress, however, they’re more likely to engage with it in the hope of improving their grade (or their learning–yes, we have those students, too).

  • Consider commenting early on a chunk.  If you have a paper due at the end of the semester, you might give some feedback on the thesis or introduction or give some feedback on a specific issue, such as organization.  You might collect a paragraph and note all the grammar errors that you expect to be eliminated throughout the final draft.
  • Consider making comments on the final paper only if students express interest.  Some professors ask for a self-addressed stamped envelope as a commitment.  Others schedule conferences (if you explain that there’s no penalty in just wanting a grade, a lot of students won’t sign up) and hand back the paper then or just comment orally.

You still have to grade, but at least you won’t feel as if you’re wasting your time casting pearls of wisdom that no one will bother to pick up.

Writing (Teaching) Tip: Teaching Students to SCROL

Yes, with one L.

I’ve been reading about teaching reading to second-language students, and this little exercise using headings and subheads in a text might benefit any student.  It’s quick to teach and focuses on the organization of a text in a way that’s useful to both readers and writers.

It’s called SCROL (Grant 1993, as described in Tricia Hedge, Teaching and Learning in the Language Classroom, 196-197).

S–Survey the headings before reading.  Ask for each one: What do I know about this topic already?   What information might this section contain?

This draws on students’ background knowledge and gets them actively engaged in reading through prediction.

C–Connect the ideas in the headings.  Ask: How do they relate?

Students begin to see skeleton of the text.

R–Read.

Students are not approaching the text cold.  Encourage marking.

O–Outline.  Outline major ideas and supporting details in each segment.  To check how well you remember the article, write the headings on a separate page and try to recreate the outline without looking at the text.

L–Look back.  Check remembered outline against text and fill in gaps.