Essay Robo-Graders Reward Form, Not Content

If you were appalled by the recently released study results that computers can score test essays as well as humans, take heart.  Machines are fast, but they’re superficial.

As NY Times reporter Michael Winrip notes in “Facing a Robo-Grader?  Just Keep Obfuscating Mellifluously,” ETS’s e-Rater can grade 16,000 essays in 20 seconds.  But as an MIT’s director of writing discovered, the computer falls for cheap tricks–big words, long sentences, empty transitions, and outright lies.

Instructors of writing still have to face the stack, but at least they have some job security.

Update on “The 7 Deadly Sins of Student Writers”

According to Ben Yagoda, punctuation is this generation’s downfall.  See his article in the Chronicle of Higher Ed, “The Elements of Clunk.”

 

Take an Editing Course from the Masters

From the birthplace of Chicago style….

The University of Chicago Graham School of Continuing Liberal and Professional Studies offers individual courses and a certificate program in editing.  Many of the courses meet for a two- or three-day seminar followed by take-home assignments.  Courses include various levels of manuscript editing (often taught by journal editors) and electives like “Editing Electronically: Using Word Processing Tools.”

Teaching Tip: The past is always with us.

This week’s principle from the Lifelong learning at Work and at Home website focuses on prior knowledge:

New information learned depends heavily upon prior knowledge and experience.

This principle stresses the importance of getting to know our students so we can help them learn more effectively.  From infancy onward, learning is based on building new mental connections that physically change brain structure.  Our brains are not built to remember unconnected facts; if material doesn’t relate to something else that is important to us, we forget.  Not only do we need prior experiences as an anchor, but the quality of our prior assumptions, conceptual knowledge and biases can all influence what we learn, for better or worse. Despite these well known findings, most of us do little to discover what our students already know (or think they know) about our disciplines. And yet, that prior knowledge may make or break their chances for success in our classes.

Why is prior knowledge so important?  Studies comparing novices and experts in a variety of fields suggest that prior knowledge is vital to the ability to access and use what we know. For example, chess experts are able to remember meaningful patterns of chess pieces much better than novices.  However, when asked to remember the positions of randomly placed pieces, experts performed no better than novices.  When the information was meaningful, the chess experts were able to “chunk” information (i.e. organize and classify it) much more efficiently than novices and then remember those larger chunks.  Instead of seeing a certain number of pieces on the board in certain places, experts see a classic opening move and relate that information to their extensive prior experience with opening moves.

How can we help students develop more effective knowledge structures within our disciplines?  Our strategies depend on the students’ current developmental level, both generally and in the context of specific disciplines.  In introductory courses, students generally have very limited ways of understanding and organizing knowledge.  But they do have life experiences, and these are important in making those first connections.  That’s why many skilled lower level instructors spend a lot of time helping students relate what they are learning to the world around them and their existing interests.  A student may not know much about biology, but she knows that everyone wants her to use hand sanitizer all winter.  From this simple observation, a series of questions naturally presents itself that can be used to build understanding.

In introductory courses we typically don’t find (or expect) students to show a sophisticated grasp of disciplinary concepts.  Unfortunately, we often find something more difficult to change: a mental framework that’s a bit dented or missing critical pieces. Misconceptions and incorrect information can distort and limit student learning, especially at the introductory level.  Unfortunately, since this incorrect information is also anchored in prior knowledge, it can be resistant to change.  Discovering common student misconceptions and designing experiences that challenge them is a critical part of building new levels of expertise.  Experiments, demonstrations, videos and other active methods that directly challenge student misconceptions are often the most powerful since they use multiple channels and can have more emotional impact than lecture or readings.  It takes a powerful stimulus to dislodge embedded rust.  However, experience is best when it is paired with explanations and principles to help students organize their new experiences effectively.  Or, as another of the core learning principles put it: Experience alone is a poor teaching. 

As students advance in the discipline, they begin to develop their own knowledge structures. In these upper level classes it’s important to find out what students already know so that you don’t try to build on knowledge that isn’t there.  Having a good understanding of prior knowledge can also help you advise students – someone with gaps that are just too large may need to take a pre-requisite course, while others may need to be referred for tutoring in specific areas.  Other students may be able to skip some topics, or take a more in-depth approach.  There are many ways to assess prior learning.  Some faculty members assess prior knowledge using pre-tests or writing assignments that identify strengths and weaknesses. A drawback of testing or writing assignments of course is the time it takes to read and analyze them, even though they are typically ungraded.  Asking students to draw a concept map of important content is a quick way to show you what students think is important and also gives you a picture of how they organize that information.  Another approach is the Knowledge survey.  This type of survey is often quite lengthy, but students are not actually asked to answer the questions as they would be on an exam.  Instead, they rate their level of knowledge of each concept or process on a three point scale from absolute certainty to complete ignorance.  These surveys can be scored electronically and they provide a quick snapshot of the class that can guide you to focus your time in class more productively.  Administering the same survey at the end of the course provides a check up on how effectively you were able to reach your goals; ideally you will see upward movement for the class as a whole and for individual students as well.

The importance of prior knowledge is also evident when we discuss transfer of learning. Many students can repeat information or use it in similar situations but, unlike experts, they may not recognize appropriate but unfamiliar applications of a concept or procedure.  The ability to recognize when and how prior information can be used in new settings is the key behind transfer of learning and also depends on how knowledge is structured in the brain. Direct instruction in relating features of the new environment or situation to the prior one can build a path to transfer, along with a lot of guided practice. Thus, presenting students with varying situations you may have to first cue the students to apply what they know, and then help them learn to recognize cues for themselves.

Above all, it’s important to realize that students’ prior knowledge and their methods for organizing it are very dissimilar from your own.  Not only did they grow up in a different world (just check the Beloit College Mindset if you doubt that) but they have not had the wealth of training and experience in your discipline that you do.  Many of us struggle with getting our minds back to that beginner stage so that we can think like students and anticipate where they need help.  If you’d like to develop that very important sense of empathy, take a challenging class in something completely new to you.  You’ll be amazed to discover how much you attempt to use your prior knowledge to anchor new material and how many misconceptions you may have!  Plus, you will experience both the frustration and the exhilaration of making progress.

Next up:  we will finish this series with the final principles of active learning, less is more and choosing what to forget.

Teaching Tip: Variety is the spice of learning

This week’s post summarizes and comments on two closely related principles from the Life Long Learning at Work and at Home website.

Principle 2: Varying learning conditions makes learning more effortful but results in enhanced long-term retrieval.

Principle 3:  Learning is generally enhanced when learners are required to take information that is presented in one format and “re-represent” it in an alternative format.

Both of these principles emphasize how important it is to vary the conditions of learning if we want students to remember and use information once they leave the classroom.  In order to understand why variability is important, it is helpful to understand how our brains store and access the things we learn.

Essentially, humans can process information in two systems, visuospatial and auditory-verbal. Information can be stored in either one of these two systems or in both of them.  According to the most commonly accepted theory in cognitive psychology, information that has been stored in both systems is more easily recalled than information that is only stored in one system or the other, so when we ask students to process information in varied ways, they are able to use multiple cues from both systems to help them remember.

One way to vary learning conditions is to present information in both major modes. For example, a reading assignment might be paired with an exercise where students must extract information from a video, a picture or a chart.  If you are in a field that is primarily visual, asking students to read about what they are seeing provides similar variation.  One caveat here – the students have to actually use both modalities.  If the students find that they can succeed using only one method, they will naturally tend to skip the other one.  Our challenge is twofold – finding good ways to use both modes and organizing our classes so that students must use them both in order to succeed.

Varying modes of presentation has distinct advantages in a classroom with diverse learners.  We all have cognitive strengths and preferences – some of us prefer auditory learning while others are visual (and still others prefer a kinesthetic approach, but that’s another story). Research on these learning style preferences suggests that trying to personalize instruction based on individual learning style does not enhance learning, and when you consider the dual storage theory, it makes sense that using multiple presentation modes with all students will provide the kind of variability that leads to increased effort and storage in both systems.  Plus, when multiple styles are used, everyone has the chance to use both their preferred and non-preferred styles.  This can help students who are non-traditional learners since it gives them a shot at the material using their preferred style as well as practice that can improve their non-preferred skills.

Not only can you vary the presentation mode (the input channels, if you will) but you can also vary the output channels to enhance retention and learning transfer.  Asking students to draw pictures or create graphical representations such as concept or knowledge maps that summarize the main points of a reading assignment or lecture works well for this purpose.  Research suggests that graphical representations are particularly useful because they force students to think about the types of relationships between concepts and information.

Assigning a concept or knowledge map exercise is most helpful if students have some prior instruction on how to complete the task. O’Donnell, Dansereau & Hall (2002) provide a good overview of knowledge maps and how to use them.  You can ask students to construct their own knowledge maps or you can give them knowledge maps instead of or along with texts and lectures.  Research indicates that giving students knowledge maps that you have constructed can help students (especially weaker students or non-native speakers) grasp material more effectively.  Giving your students pre-constructed maps might be more appropriate for less advanced classes, while students in advanced classes can be challenged to produce their own.

In addition to concept and knowledge maps, arguments and problem solving procedures also are good candidates for visual diagramming methods.  The University of Texas Center for Teaching and Learning presents a simplified version of diagramming arguments while this pdf presents a more detailed and formal  version drawn from philosophy.

Principle 2 above also indicates the downside (at least from the student perspective) of using variable learning conditions – the need for more effort.  When students are asked to learn material under varied conditions or “re-present” material in a different format, as in Principle 3, they have to work harder. The bottom line is obvious to the point of being somewhat trite.  When learning requires more investment of effort, it is more likely to be retained

Because of the increased effort required, students may seem to learn more slowly when you vary the conditions of learning.  Don’t despair and don’t give up.  When a single modality is used (e.g. readings and lectures accompanied by exams – all verbal mechanisms) both you and the students may falsely assume that they understand the material on a deep level.  Requiring students to use different methods and media for their learning may result in poorer performance initially, but the research suggests that long term learning is enhanced.

You are likely to hear from students that multimodal work is harder than traditional single-channel methods.  Validate their correct observations!  Students need to understand why you are “doing this to them.”  Sometimes students who are very able in one modality (like most college professors) are particularly resistant to trying new and challenging modes.  And many of us are reluctant to leave our comfort zone as well.  But the research is quite clear that doing so enhances learning and transfer.

What methods do you already use to vary the conditions of learning in your classes?  What would you like to know more about? 

If you want to read further:   

The Lifelong Learning at Work and at Home website provided these summarizes and recommends these articles for additional reading

Mayer, R. E. (1993). Illustrations that instruct. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 4, pp. 254-284). Hillsdale, NJ: Lawrence Erlbaum Associates.  This book chapter explores the uses of text and illustrations as teaching aids, primarily in textbooks.  The author examines how different types of illustrations (i.e., decorative, representational, organizational, and explanative) affect cognitive processes—selecting, organizing, or integrating information—that are involved in learning.  Explanative illustrations show how elements in a system are related and underlying principles governing the system.  Although underused in textbooks, these types of illustrations best promote all three types of cognitive processing that enhances learning.

Meyer, B. J. F., & Poon, L. W. (2001). Effects of structure strategy training and signaling on recall of text. Journal of Educational Psychology, 93, 141-59.  Training older and younger adults to use textual cues that highlight conceptual relationships improved their overall recall of the text as well as recall for main ideas.  Training produced positive transfer to remembering everyday materials such that these individuals also better recalled details from informative videos, relative to individuals who were given motivational training or no training.

Wallace, D. S., West, S. W. C., Ware, A., & Dansereau, D. F. (1998). The effect of knowledge maps that incorporate gestalt principles on learning. Journal of Experimental Education, 67, 5-16.  Learning aids were presented in one of three different formats: text, unenhanced map, and enhanced map.  The enhanced map differed from the unenhanced map in that it used the gestalt principles of similarity and proximity to group related concepts.  Those who studied using enhanced maps demonstrated superior recall over those using unenhanced maps or text.

 

References

O’Donnell, A.M., Dansereau, D.F. & Hall, R. H. (2002). Knowledge maps as scaffolds for cognitive processing. Educational Psychology Review, 14 (1), 71-86

 

Coming up next:  The importance of prior knowledge to present learning.

Use Cognitive Research to Enhance Teaching: Practice at Retrieval

Welcome back!  One of my New Year’s resolutions is to get the blog up and running again, so I hope this post will be the first of several this semester.

As scholars, we are always looking for high quality research related to our disciplines and the courses we teach.  Why not apply that lens to learning?  The field of cognitive psychology has developed dramatically over the past 20 years or so, and we now have quite a large body of research describing how people learn.  I don’t want to overwhelm you with data, so I am going to create a series of posts that describe nine primary principles of learning and how they apply in our classrooms.

The nine principles and some of the examples in these posts come from the Lifelong Leaning at Work and at Home initiative website.  This initiative was started by a group of cognitive researchers dedicated to applying cognitive science research to lifelong learning and higher education.   The website has detailed links to more in-depth information and references if you would like to dig deeper.

This just means that students need multiple opportunities to recall and use information or skills that you want them to remember after your class is over.  If students merely re-read notes or books without practicing retrieving what they have heard and read, they will not retain that material for long.  My first reaction to this statement was “well, duh” but the “single most important” label made me pay more attention, as well as the mention of transfer.

If this statement is correct (and there is quite a lot of research to back it up) what does it imply for the classroom?  What can we do to help students practice retrieving important information?  Here are a few possibilities:

  • Align your classroom activities, assignments and tests so that students must repeatedly access the same information.  For example, ask students questions that require them to recall and demonstrate understanding of material from earlier in the course.  You may want to pose a probing question and have everyone write down a response before you ask for volunteers – that way the entire class gets to practice retrieving information.  Otherwise, only the individuals called on will be engaging in this important task.  This technique also allows you to correct any misunderstanding and it shows students the cumulative nature of learning.  At first, you may get blank stares but if you do this consistently students are more likely to get the message.
  • Test repeatedly on the same material.  My undergraduate self hated the cumulative final exam with great passion, but it is an excellent method for promoting long term retention and retrieval – but only if the material has already been tested earlier in the semester.   You also can get this effect by using chapter quizzes and then repeating important material on exams.  Even giving a unit exam on the last day of class and then a cumulative final a week later will help with retention of recently presented information.   Research suggests that spacing the testing out across the semester leads to better results, and that for maximum effect recall should be somewhat delayed.  As we have probably all experienced, testing or recall efforts that occur immediately after teaching or reading material tends to produce short term positive effects that disappear quickly.   So you might want to start by giving a reading quiz perhaps a week after the reading was discussed in class.  Questions on the same material could appear on a mid-term or be incorporated into a later assignment and then tested again on a comprehensive final.  For maximum effect, the student should be using recall methods like short answer questions or essays and not recognition methods such as multiple-choice, true-false or matching.
  • Encourage students to question themselves or each other instead of re-reading notes or texts.  Give a series of open-ended questions as a study guide or have students bring open-ended questions to class, exchange them, and practice answering them.  Online quizzes can work as recall practice too, although they tend to be more recognition focused.  Assign online quizzes strategically to keep the students repeatedly working with the material over the course of the semester.
  • If you do not use tests, you can still require students to recall and reuse previously learned material for projects, case studies or other activities.  Varying the method of retrieval e.g. using an in-class exercise or presentation instead of a test, enhances retention, since it gives students multiple cues for recalling information.  Material that becomes embedded in a narrative or other rich experience is more likely to be retained (but more on that later).
  • When asking students to retrieve previously learned material, try to provide as few hints (“retrieval cues” in cognitive jargon) as possible.  Thus, a free response essay or an application that requires the student to recall material is better at promoting retention than a multiple choice question that requires only recognition of the correct answer.
  • Give students immediate feedback on their answers to avoid them practicing and learning incorrect material.  When you are working on this kind of long term learning, you want to make sure it is correct!

While these ideas and suggestions can help us design courses that maximize students’ ability to remember and transfer information more effectively, we still have to decide which material needs to be emphasized in this way.  We don’t have the time to require frequent testing and recall of all or even most of the material in a typical course, so it is vital for us to distinguish between material that must be automatically available versus material that can and should be looked up as needed.  The current wealth of easily accessible online information has made this a difficult question, but looking at basic conceptual frameworks, core concepts and strategies is a good place for most of us to start. Using a cognitively informed approach asks us to be more intentional about identifying and choosing the most important material in the course and strategically requiring students to recall it multiple times in multiple ways.  It may require rethinking some aspects of your course, but the reward is longer retention and better transfer of your course’s most important concepts.

Most of this post has been summarized and paraphrased from http://psyc.memphis.edu/learning/principles/lp3.shtml

Next up:  Varying Learning Methods

Teaching Tip: Better Student Study Skills

An article about the National Survey of Student Engagement (NSSE) in the November 17 Inside Higher Education grabbed my attention with a table showing the use of a variety of learning strategies among freshman and senior respondents.  I think the results provide some insight into student study patterns and some useful guidance for instructors.

Not surprisingly, the most commonly endorsed strategy was “taking careful notes in class” which was endorsed by 88% of freshmen and 86% of seniors.  Unfortunately, many students were not doing much with those notes after taking them (and many of us may wonder how careful those notes really are).  Only about two-thirds of each group said they reviewed their notes after class, and only about three-quarters of either group went back and organized their notes to make them more meaningful.

When it came to completing reading assignments, around 80% of both groups reported that they “identified key information” in their reading.   However only two-thirds reported that they stayed focused while reading course and avoided distractions during studying. Sixty percent or fewer of the students who responded took notes while reading.   How effectively can they be identifying key information if they are distracted, unfocused and not taking notes?

More active strategies were used by even fewer students.  Only about half of those responding said they created their own examples to help them study, or outlined major topics or ideas from their study materials.

Finally, while three-quarters of the students surveyed said that they “set goals before starting academic tasks”, fewer sought help when they did not understand, and only about half of freshmen and fewer than half of seniors ever discussed effective study strategies with anyone.  The only other strategy that was chosen by 80% or more of both groups was “connected to learning things you already knew”.  This is an excellent strategy, but one wonders at what level this is happening given some of the other responses.

The clear implication of these findings is that our students really don’t know what it means to study effectively and efficiently.  For decades, research in cognitive psychology and education has demonstrated the need for learners to actively engage with material if they are going to truly learn and remember it.  If students are not spontaneously adopting active study strategies, we can help them by building these strategies into our course requirements.  Most of us do try to help students link new knowledge to prior knowledge, but there are more concrete things we can also do.  For example, requiring some kind of written response to readings such as notes, response logs or responses to questions based on course readings helps students learn to identify what is important and to concentrate while reading. Asking them to create study guides that involve outlining, organizing and/or summarizing their class notes and readings will promote the kind of higher level thinking that we are trying to foster. When students create case examples or problems, write potential exam questions, or otherwise actively use course material, they are more likely to develop habits of deep studying that lead to deeper learning.

Fine for freshmen, perhaps, but shouldn’t upper level or graduate students be able to do this on their own?  At least according to these NSSE results, the differences in responses between freshmen and seniors was minimal, which suggests that students are not internalizing effective study strategies as they progress through undergraduate education.  It also suggests that beginning graduate students are unlikely to have highly developed study skills.  Some schools have attempted to address the issue by requiring study skills, orientation and/or critical thinking courses or workshops, but these tend to be ineffective unless the skills are somehow embedded into a larger topic or them.  Robert Leamson quotes the response of a failing student when asked why he was not using the success strategies that he had learned in a freshman orientation course.  “You mean we were supposed to actually do all the stuff in that book?”  (Leamson, 1999, p. 41)    I think that by working these skills into many of our course requirements we are likely to reach a much broader audience as well as ensure that they “actually do” what we know works.

What strategies do you use to help students develop better learning habits?

To read the entire article Major Engagement (which addresses a number of other interesting findings and issues from the NSSE) go to

http://www.insidehighered.com/news/2011/11/17/nsse-2011-measures-student-engagement-major?utm_medium=facebook&utm_source=twitterfeed#ixzz1dzE5czJM

Leamson, R. (1999) Thinking about teaching and learning.  Sterling, VA:Stylus.