Automated feedback: The new science of grading STEM papers

Science educators at Wake Forest University are testing how automated feedback combined with new one-on-one teaching methods can improve scientific writing from STEM undergraduates – and result in better explanation of research to the public.

The National Science Foundation (NSF) has awarded the research team nearly $280,000 to conduct the writing experiment. It comes at a pivotal time, with claims of junk science and fake news making headlines almost daily.

“Students don’t see that writing is important in the sciences,” said Dan Johnson, who leads the research team and is a teaching professor in the Biology Department.

The STEM (science, technology, engineering and math) writing study will build on an automated lab report feedback system called SAWHET (pronounced saw-it). Undergraduate science students have been using SAWHET to submit and revise lab reports online for two years.

SAWHET (or STEM Automated Writing Help Tool) guides students to break down their lab reports into the proper sections of a scientific paper – title, extract and citations, for example. Help buttons explain those sections, and the software can ask some basic editing questions: Do you need a citation to back up this statement? Do you state your hypothesis? Once the writer addresses those questions, the report is re-submitted.

With the NSF grant, the researchers will teach SAWHET to ask additional content-focused questions, allowing instructors and teaching assistants to concentrate on providing deeper, one-on-one feedback.

Time has been the primary challenge to teaching writing more aggressively to STEM students. One college course can include 150 students. If the instructor gives a writing assignment and edits it twice for about 15 minutes each, that can add up to 70 hours of work.

But writing and communication skills are increasingly important to employers. A report by the Association of American Colleges & Universities found that 80 percent of employers surveyed wanted colleges to emphasize written and oral communication skills more.

“Employers are often complaining that writing skills are poor,” said Sabrina Setaro, program director of the STEM writing project.

That’s why the Wake Forest study also will introduce new methods that instructors will use for teaching and evaluating students’ scientific writing. Instructors will be encouraged to let SAWHET do the low-level proofing and format checking, so they can do a deeper dive into the logic, arguments and thinking behind the written words.

“We’re seeing places where people are using less effective practices because it’s familiar – but we’ve got evidence that shows that doesn’t help a student progress,” Setaro said.

This isn’t the first time Johnson and Setaro have flipped the undergraduate science curriculum. They both work on BioBook, a collaborative, interactive online biology resource that replaces traditional written textbooks and encourages inquiry-based learning. The basic edition is free; a customizable version is available, for which students pay $30 per year for access. About 750 students and professors at two universities are using it right now.

SAWHET is an offshoot of BioBook.

“Students were using the assessments in BioBook, and we realized that writing was a major barrier for them,” Johnson said. “We wanted to give students a chance to assess their writing, too.”

Johnson and Setaro, with co-investigator Jerid Francom, associate professor of Spanish and linguistics, will use the first two years of SAWHET data as the control, and will begin collecting new data in the spring 2018 semester. Different stages of collection will reflect different levels of feedback students receive from both SAWHET and instructors.

“Improving Scientific Writing in Undergraduate STEM Classrooms: A Training Program for Students and Teaching Assistants Aided by Information Extraction Technology” runs through July 2020.