Automated feedback: The new science of grading STEM papers

Wake Forest biology professors Dan Johnson and Sabrina Setaro talk with graduate students Noah Bressman and Jenny Howard about new methods for grading science writing, in a lab in Winston Hall.

Science educators at Wake Forest University are testing how automated feedback combined with new one-on-one teaching methods can improve scientific writing from STEM undergraduates – and result in better explanation of research to the public.

The National Science Foundation (NSF) has awarded the research team nearly $280,000 to conduct the writing experiment. It comes at a pivotal time, with claims of junk science and fake news making headlines almost daily.

“Students don’t see that writing is important in the sciences,” said Dan Johnson, who leads the research team and is a teaching professor in the Biology Department.

“They don’t see much value in it. They don’t realize we actually live and die by our writing.” Dan Johnson

The STEM (science, technology, engineering and math) writing study will build on an automated lab report feedback system called SAWHET (pronounced saw-it). Undergraduate science students have been using SAWHET to submit and revise lab reports online for two years.

Defining STEM writing

This document is an example of a lab report submitted by a student using the SAWHET automated system at Wake Forest University.

This is not a perfect report, however. Dan Johnson, in the Department of Biology explains:

“This would be a B-grade report by our current standards. The main parts are all there, but there are some mechanical errors and oversimplified thinking. The language also is a bit confusing. If I were the teaching assistant, I could assign a grade to this report in three to five minutes, then spend most of my time making substantive coaching comments about how to improve overall rather than counting points and correcting typos.”

SAWHET (or STEM Automated Writing Help Tool) guides students to break down their lab reports into the proper sections of a scientific paper – title, extract and citations, for example. Help buttons explain those sections, and the software can ask some basic editing questions: Do you need a citation to back up this statement? Do you state your hypothesis? Once the writer addresses those questions, the report is re-submitted.

With the NSF grant, the researchers will teach SAWHET to ask additional content-focused questions, allowing instructors and teaching assistants to concentrate on providing deeper, one-on-one feedback.

Time has been the primary challenge to teaching writing more aggressively to STEM students. One college course can include 150 students. If the instructor gives a writing assignment and edits it twice for about 15 minutes each, that can add up to 70 hours of work.

But writing and communication skills are increasingly important to employers. A report by the Association of American Colleges & Universities found that 80 percent of employers surveyed wanted colleges to emphasize written and oral communication skills more.

“Employers are often complaining that writing skills are poor,” said Sabrina Setaro, program director of the STEM writing project.

“Writing helps students be concise, order thinking, create a story and build an argument.” Sabrina Setaro

That’s why the Wake Forest study also will introduce new methods that instructors will use for teaching and evaluating students’ scientific writing. Instructors will be encouraged to let SAWHET do the low-level proofing and format checking, so they can do a deeper dive into the logic, arguments and thinking behind the written words.

“We’re seeing places where people are using less effective practices because it’s familiar – but we’ve got evidence that shows that doesn’t help a student progress,” Setaro said.

This isn’t the first time Johnson and Setaro have flipped the undergraduate science curriculum. They both work on BioBook, a collaborative, interactive online biology resource that replaces traditional written textbooks and encourages inquiry-based learning. The basic edition is free; a customizable version is available, for which students pay $30 per year for access. About 750 students and professors at two universities are using it right now.

SAWHET is an offshoot of BioBook.

“Students were using the assessments in BioBook, and we realized that writing was a major barrier for them,” Johnson said. “We wanted to give students a chance to assess their writing, too.”

Johnson and Setaro, with co-investigator Jerid Francom, associate professor of Spanish and linguistics, will use the first two years of SAWHET data as the control, and will begin collecting new data in the spring 2018 semester. Different stages of collection will reflect different levels of feedback students receive from both SAWHET and instructors.

“Improving Scientific Writing in Undergraduate STEM Classrooms: A Training Program for Students and Teaching Assistants Aided by Information Extraction Technology” runs through July 2020.

Categories: Faculty, Research, Top Stories

Wake Forest News

Meet the News Team