Research Auditor Simulacrum

What The Tool Does

The Research Auditor reads a research paper — or a draft paper, or a summary of a paper's central claim — and assesses whether the methods used can support the claims made. It examines the fit between research question and study design, the appropriateness of the sample, the adequacy of the statistical or interpretive framework, the control of confounding variables, the honesty of the reported uncertainty, and the distance between what the paper demonstrates and what it is presented as demonstrating.

The output is a structured critical appraisal: a description of what the paper actually claims (which is often not identical to what its abstract claims), an assessment of the method's adequacy for that claim, an inventory of the threats to validity that the paper addresses well or poorly, and an overall judgement of how strong a contribution the paper makes. Where the claims exceed what the method can support, the auditor says so specifically — what claim the method can support, and what gap the paper's stronger claim requires additional evidence to close.

Where The Method Comes From

Systematic critical appraisal of research is an older practice than most of its formalisations. Every rigorous reader of a scientific paper has always been doing it informally. The formalisation — the development of explicit appraisal frameworks, scoring systems, and checklists — dates mainly to the evidence-based medicine movement led by David Sackett, Brian Haynes, and others at McMaster University in the 1980s and 1990s. Sackett's teaching that every reader of a clinical paper should ask "Are the results valid? What are they? Can I apply them?" is the foundational template for the auditor's three-stage examination.

The tools of critical appraisal have since proliferated across disciplines. The CONSORT statement for randomised trials, the STROBE statement for observational studies, PRISMA for systematic reviews, the Cochrane Risk of Bias tools, the Joanna Briggs Institute frameworks, and the various discipline-specific reporting guidelines all implement the same basic move: decompose a paper's claims into checkable components, examine each, and arrive at a judgement. The auditor inherits this toolkit and adapts it to papers across the sciences, social sciences, and — with appropriate modifications — the humanities.

What It Can And Cannot Do

The auditor can appraise papers across most empirical disciplines, identifying methodological weaknesses, threats to validity, gaps between claim and evidence, and instances of overclaiming. It is particularly useful for researchers evaluating papers they intend to cite, for students learning how to read critically, and for authors stress-testing their own drafts before submission.

It cannot replicate a study or verify its data. It works on what the paper reports; if the paper misrepresents what was done, the auditor cannot detect that directly. It also cannot adjudicate between rival interpretive paradigms in fields where methodological disagreement is itself the substance. It reports what any competent reader of the relevant literature should notice, and leaves deeper judgements to the reader.