AI Plagiarism Considerations Part 1: AI Plagiarism Detectors

AI Plagiarism Checkers--they're not our friends...

This is part of a three-part series (here is part 2 and part 3). In this post, I’ll talk about navigating the discussion of AI plagiarism-checkers. The second one will discuss ways of engaging with students when AI-plagiarism is involved, and finally, I’ll end with recommendations on “the talk” that faculty should be thinking about having with students at the start of a class

Image source: ChatGPT 4

Regularly, I get an email like this from faculty I work with, colleagues, or even folks I don’t know but whom are familiar with my work:

“Hey Lance,

I got a student that I know used AI for their assignment. What do you recommend for addressing this?”

First and foremost, I appreciate that folks are taking the time to double-check their own thinking about what’s the best course of action. I know (from my own experiences), how easy it can be to have a knee-jerk reaction to believing I have discovered some deception (from students or anyone for that matter). It can trigger a reaction when you believe you are promoting a trusting environment. To take the time to check one’s self is incredibly important because in such moments, we’re often not just reacting to the specific incident but super-imposing previous instances of perceived deception into this moment (related or unrelated to the context).

So my next moves are a little tricky. Because I have to ascertain what they mean that they “know” that a student used AI. When it comes to AI usage, we often suspect but do not know in a way that is actually fact. We can know that the student’s work is drastically different from what has come before. But they don’t know that student used AI.

So in my initial response, I usually try to learn how they have determined the usage of generative AI in the student’s work. I’ll say something like, “Out of curiosity, how did you determine that a student used generative AI in submitted work? I ask, in part, because I know determining for sure that AI has been used can be a challenge and I’m regularly looking for clear ways to establish proof that I can share with others.”

I am genuinely interested in finding ways that require proof and also, by framing it this way, it helps faculty to slow down a little bit. It raises some questions and considerations about what they do actually know and what they suspect. It’s often in this moment that they reflect back that they don’t know for sure but a student’s work is looking drastically different in some way.

Sometimes, they explain that they just know—from their own professional expertise as educators or from how drastic the student work has changed from other works by the student. If they don’t indicate plagiarism checkers, I skip the part about AI plagiarism checkers and go into offering the strategies for conversation with the student (which you can read after the next section).

If they say they know based upon how it sounds though, I do try to remind them there’s a feedback loop happening on different levels to beware of. The first is that everyone’s radar is up looking for it and knowing that it exists. This can cause us to scrutinize writing in a way we might be over-assigning some kinds of writing to AI. So we need to be careful of that.

Furthermore, we have to double-check our own clarity about usage. What if the student wrote a draft and asked AI to revise or clean it up? Where does that show up as a policy in the work? In that context, it may sound like AI and also be OK because it wasn’t clear such things were allowed. And if you’re asking for grammatical and style expectations and allow for spell & grammar checkers, have you clarified what that means (or really made a clear explanation of why having AI do this instead of Grammarly’s non-AI or other tools do this is a difference with distinction?

Explaining AI Checkers to Faculty

Most of them time, folks tell me that they have used AI plagiarism checkers to verify it. Some will explain they tried several to verify it. And that’s when I share the following in order to help them understand the challenges of using Generative AI plagiarism checkers.

I thank them for sharing and explaining that they used an AI plagiarism checker. If it’s at my institution, I do remind them that using AI plagiarism checkers is actually counter to our policy.

Now, if we’re having a live conversation, this is easier to do but if it’s email, then I know I have to take a different tactic. Back and forth chains can take a long time and the clock is ticking on the semester. I start with explaining that I realize they may be aware of the following and apologize if I’m over-explaining but it's not always clear what folks know/don't know about them.  I then say some variation of the following:

Because the AI detectors are poor indicators, it is important to check first with students (for instance, I know plenty of writers who have put their work into AI checkers and gotten back that their work is 100% AI generated).  Also, the student may have written up something and then used AI as the grammar/style improver.  That might not be ideal or be concerning but that's different from a 100% AI generated paper--yet the AI plagiarism generator isn't going to distinguish such things. 

AI detectors have significant flaws and in the end, they are probability without evidence.  That is when it registers that something is 100% AI, the checker does not have actual evidence of AI usage but the work has a pattern of language that is 100% similar to how AI generates language--except of course, AI-generated language is language patterns created from studying human language patterns--so there is a bit of negative feedback loop there.

How AI plagiarism checkers calculate plagiarism is different in nature from traditional plagiarism detectors that can match work from a particular source.  They can point to something that exists and say, “it looks like this.” The AI detector is merely saying it certainly could be AI generated--but it has no actual evidence.

One of the big challenges with AI detectors is that it can and does create false positives.  And the research on AI detectors show that it is more likely to produce a false positive for students for whom English is a second language (because we often learn language in predictable patterns that we reproduce as we make sense of the language's nuances).  

Coupled with that, there's concerns about privacy and ownership of putting students' work into other programs that are likely using it as part of its own user data and possibly profiting from that.  That is, do we have the legal right to hand over the intellectual property of students over to these tools that will then profit from them?

This means that to say we have proof isn't accurate.  We have probability and unfortunately, we have probability that is impossible for the student to disprove since we don't get to actually look under the hood of the AI detector. That detector is entirely detached from any one of the hundreds of AI tools out there (i.e. the AI detector can't say--"look, here's the same output from this AI tool on this date/time") and cannot actually know or prove anything. So in any situation, we're taking a gamble on a machine that has no obligation (financial or legal) to help us with the actual truth and pitting it against the student.  

That feels a little tenuous of a place to be with a student—especially if we’re trying to foster a space for learning and trust. Using their work to profit other companies and then to reflect back on them a probability without evidence of their supposed cheating. In this way, a false positive can do irreparable harm to a student and their education. So we want to move thoughtfully forward.

(Side note: I don’t typically include this in the moment but one of the reasons I grow weary of accusing a student without evidence is my own experiences in my younger years of being accused of doing things without evidence that I didn’t do. The label theory was real even before I knew it. After being falsely accused, I did end up doing the things I was accused of because I figured “well, if that’s what they think of me, I might as well.” I’m not saying every student will respond this way but yes, we will lose some of whom we accuse).

What’s An Instructor To Do?

Again, none of this means we shouldn't pursue what might be an anomaly in a student’s work. And yes, we want to hold the student accountable if they had their work done by some other means than the clearly stated ways of doing it. But we have to do it by being in conversation with the student.  Ideally, the conversation should be more discovery and observation of facts.

And this is the hardest part. AI plagiarism checkers are so seductive with their matching and scoring. It makes it clear, it gives it the veneer of precision, it gives us a sense of certainty. And it’s all bullshit. It allows us to bypass the actual work of teaching, which can also be a bit of detective work—figuring out what students really have learned and not learn; navigating the gap between what they understand and the limited means of expressing their understanding.

But we have to use our curiosity and step into the conversation with students about their work. So for instance, I encourage faculty to take an approach like this when speaking with students: "Your writing in these were quite distinct from what you've written before.  I'm curious to know what was different this time, both to understand your process and to see if there is something that I can leverage."  

Or "this seems at distinct from other work, can you walk me through your process"--that is, a conversation to "show their work" and make openings for them to share what they used or even, something like, "I've seen similar answers from students in the past and when I inquired, they shared that they used ChatGPT to help them.  Would this be a situation like that?". That last one I would deploy carefully and thoughtfully as the conversation transpired.

The goal here is to actually dig further into learning about the student’s process and pinpoint why this work might be different—but to do so in a way that is exploratory and not dismissive or necessarily with the end goal of trying to “catch them.”

That’s all for this post—in the next one, we’ll talk about what happens when students say or don’t say they used it.

---

AI+Edu=Simplified by Lance Eaton is licensed under Attribution-ShareAlike 4.0 International