The Silent Crisis of Bad Data in Graduate Research

I still remember the email. It came in at 11:47 p.m., subject line "quick question before submission," from a student I'll call Aisha, though that isn't her name. I'd been reading drafts of her thesis for eight months by then, and I knew this document the way you know a piece of furniture you helped build, every joint, every place it had been sanded down and reinforced. Her theoretical framework was, genuinely, some of the sharpest work I'd seen from a master's candidate that year. She'd gone back and forth with her committee three times on it and won every argument on the merits.

The email had one attachment: a screenshot of a SmartPLS output. Her question was short. "Does this look okay to submit?"

I looked at it for a long time before I answered, because the honest answer was that I had no idea, and neither, I suspected, did she. The path coefficients were there. The R-squared value was respectable. Nothing was flagged in red. But when I asked her, over a rushed phone call the next morning, why she'd chosen a reflective measurement model for one particular construct instead of a formative one, there was a pause that told me everything. She'd chosen it because that was the default the software offered, and the default hadn't complained.

She wasn't lazy. She wasn't careless. She was two days from a deadline, running on four hours of sleep a night for a week, and she had done exactly what the culture around her had trained her to do: pour everything into the argument and the writing, and treat the statistics as a machine you feed data into and trust to hand back the truth.

That phone call is the reason I wanted to write this.

The chapter nobody argues about

Here is what struck me most, sitting with Aisha's file open on one screen and her citation manager open on the other. She had fought, really fought, for every theoretical claim in her literature review. She'd pushed back on a supervisor who wanted her to cut a construct she believed mattered. She'd re-formatted her entire reference list twice because she couldn't decide between two acceptable variations of APA spacing, and she'd asked me, unprompted, whether a comma before "and" in a three-item list was still correct in the newest edition.

She had never once asked whether her sample size was adequate for the model she was running. She had never asked what a poor discriminant validity result would actually mean for her conclusions. Those questions simply hadn't occurred to her as things worth fighting over, because nobody around her, not her cohort, not the workshops, not the how-to videos she'd watched, had framed them as arguments to be won. They'd been framed as steps to complete. Click here, check that this number is above 0.7, move to the next section.

This is the part that keeps me up more than any individual mistake a student makes: the software has gotten so good at producing confident-looking output that it never signals when the question underneath it was wrong. SmartPLS did not know that Aisha's construct should have been formative. It just ran the model it was given and returned numbers with the same clean authority whether the underlying decision was right or catastrophically wrong. It is possible to run a statistically "successful" analysis on a completely broken premise, and nothing on the screen will tell you that's what happened.

What happens after the viva

I asked Aisha, months later, over coffee, whether she'd gone back and re-examined that model. She had, a professor on her committee had, gently but firmly, made her. It held up, in the end, mostly by luck and a sample that was more forgiving than it should have been. But she told me something I haven't forgotten. She said the scariest part wasn't the number itself. It was realizing that if her committee hadn't asked, she would have submitted it, defended it with total confidence, and never known there was anything to question.

That is not a story about one student's gap in training. I've heard some version of it from a dozen people since, a friend whose master's thesis got published in a decent journal with a mediation analysis that, on closer inspection years later, never should have cleared review; a colleague who found out during her own PhD that a foundational paper in her subfield had been quietly contradicted by three failed replications, all traceable to the same kind of assumption nobody had checked the first time around. Multiply Aisha's 11:47 p.m. email by every graduate program, every semester, and you start to understand why so much of what gets called "the replication crisis" was never really about fraud. It was about a lot of exhausted, well-meaning researchers trusting an output screen more than they trusted their own understanding of what they were doing.

And the cost doesn't stay in the thesis. It becomes the conference talk. It becomes the citation another student builds a whole chapter on, three years later, without knowing the foundation was never checked. Credibility in this field is the only thing anyone actually keeps once the degree is framed and hung on the wall, and I've watched it get quietly, permanently dented by a data chapter nobody wanted to slow down and defend.

Learning to argue with your own data

What changed for Aisha, and what I try to pass on to every student since, wasn't a better tutorial on SmartPLS. It was a single habit: before accepting any output, explain out loud, to yourself, to a friend, to me on the phone if nobody else is around, why this specific test, this specific model, this specific cutoff, is the right one for this specific question. Not what the software calls it. Why it's right.

That habit is uncomfortable in exactly the way real intellectual work is supposed to be uncomfortable. It means sitting with a result that doesn't confirm your hypothesis instead of quietly re-running the model until one does. It means being willing to write, in your findings chapter, that something didn't work the way you expected, because that's data too, and burying it is a choice, not a neutral act. Aisha's model happened to survive scrutiny. Plenty don't, and that's fine. A thesis that reports an honest null result and explains it with care is worth more than one built on a result nobody can defend under questioning.

I don't think this gets fixed with a better software workshop, though those help. It gets fixed the way Aisha's did, with someone asking "why this one, and not the alternative?" early enough that there's still time to actually answer it, rather than two days before a deadline, in a screenshot sent at midnight.

What I'd tell her now

If I could go back to that email, I wouldn't just answer "does this look okay." I'd ask her the question first: why this model? I think about how differently that email exchange goes if someone had asked her that in month three instead of month eight.

A dissertation isn't a word count with a bibliography stapled to the end. It's an argument, all the way through, including the part that happens inside a piece of software most students were never really taught to interrogate. The comma placement in your reference list will not be the thing that comes back to haunt you a decade into your career. The model you didn't fully understand, and defended anyway, because the output looked clean, might.