Can Teachers Tell if An Essay is Written by ChatGPT?

A student ponders his use of ChatGPT / Credits: Generated by the Dall E text-to-image model by David Sipos

As generative artificial intelligence (AI) programs have grown in popularity, they’ve become an increasingly viable tool for students yet pose a point of concern for teachers. Already, students are using ChatGPT to help with assignments, even if they “don’t know if the teacher would have permitted it,” according to one anonymous student. 

Teachers are increasingly aware of the prevalence of generative AI and have used tools like turnitin.com to detect its use in class. For example, humanities teacher Laura Webster uses the software for assignments that are not handwritten. 

The anonymous student noted that they found ChatGPT to be a valuable tool but did not believe it could produce work at the same level as an actual student. 

Ms. Webster agreed, stating that as a classroom tool, “AI is not good.” It is particularly unhelpful for her 12th grade students, who have to write on more complex subjects. 

To examine the difference between AI and student work, The Rookery conducted an experiment to see if teachers could determine who wrote a paper. 

Three teachers — Ms. Webster, humanities teacher Ginea Briggs, and art history teacher Jason Bulluck — were provided with three versions of an opening paragraph of an essay analyzing John Cheever’s short story, “The Swimmer.” 

One was written by a Walls AP Literature and Composition student, another is that same student’s writing edited by ChatGPT, and the last was written entirely by ChatGPT, based on the same prompt. The teachers were then asked to identify which was which. 

Mr. Bulluck successfully identified the student work, Ms. Briggs identified the AI written work, and Ms. Webster accurately identified all three. 

Ms. Webster attributes her success to the fact that she has spent a lot of time learning about and using ChatGPT and has come to understand its writing patterns better. 

“Everyone has a style,” she said, “and I know its voice.” She explained that “ChatGPT is very general in its language but has very sophisticated word choice.” 

Looking for these hallmarks allowed her to identify the AI written work and she found that she easily identified the student work as well because the way it was structured was “how you’d teach a student to structure a thesis. ChatGPT wouldn’t do that unless you told it directly.” 

Other results were more of a mixed bag, highlighting that without spending time analyzing the differences between AI and student writing, ChatGPT’s writing can be more difficult to identify. 

However, Mr. Bulluck is not particularly concerned about increasing student usage of AI. “The classroom … should reflect that AI can be super helpful, and who knows what brilliant surprises are ahead as student artists, student STEM explorers, and student writers find ways to incorporate the machines to help us towards more equality and broader understanding,” he said. 

Bulluck further noted that he sees AI as a potentially valuable tool in eliminating busywork in the classroom because “traditional editing [is] more time consuming and less direct.”

Other teachers see AI as a tool for doing just the opposite — for focusing more on editing and revision. “I think in the future,” Ms. Webster said, “AI’s presence will have us switch to caring more about editing and revision than first drafts because ChatGPT can write first drafts.” 

Ms. Webster further explained that she thinks editing will continue to be important despite AI being able to edits drafts. This is because, even if a student is using AI to revise a paper, being able to prompt the AI proves that they understand the skills that they are applying in a certain piece of writing. 

Students believe that AI is valuable in making their work more efficient. “[ChatGPT] was really helpful in a time crunch,” said the anonymous student. 

Although both Walls’ students and teachers seem to be beginning to accept some uses of AI in the classroom, school policy does not yet reflect their open-mindedness. 

Walls’ current AI policy implemented this year requires students to cite generative AI tools like they would any other source. The policy further dictates that should AI be detected in student work without a proper citation the student will be subject to typical punishments and grade reductions for plagiarism. 

The updated policy, though, is unclear as to how to cite the AI. It also does not specify whether or not AI would be acceptable in the way many students and teachers envision its use  — as an outlining, drafting, revising, or editing tool rather than as a sole author of the work. 

Confusion abounds when it comes to AI in the classroom. A Common Sense survey released in May found that nearly 60 percent of high school students had used ChatGPT or similar generative AI. As AI becomes commonplace, Walls may be forced to grapple sooner rather than later with these areas of uncertainty. Until then, teachers and students will continue to debate what constitutes appropriate use of AI.