[2026 update] Confounding by indication, explained with a concrete example

Confounding by indication, explained with a concrete example Writing it up because I had to work it out twice and would rather nobody else did.

I have seen SURPASS-4 (Lancet, 2021) cited in support of a claim I do not think it supports, twice this month, so I would like to work through what it actually shows.

My reading is that the trial is sound for its own question and is being stretched to answer a different one. I might be wrong about that, which is why this is a topic rather than a correction.

What I would like from this discussion: someone who disagrees with me to say why, with the section of the paper they are relying on.

The opening post answers the question as asked. The question underneath it is different.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

I checked the source rather than the summary, and they differ.

No disagreement from me. Posting only so the question does not look ignored.

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

Worth separating two things that the opening post runs together.

Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.

The step people skip is the one I have spelled out.

Post #5 is right about the mechanism and I think understates the practical bit.

A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.

That much is documented. The rest is how I have interpreted it.

Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.

Agreed on all of that, and I have nothing to add to it.

On post #5 — agreed on the reasoning, with one qualification.

Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.

This is the sort of exchange that makes the archive worth searching.

Post #9 is right about the mechanism and I think understates the practical bit.

Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.

Old habit: I write down the expected answer before I calculate it.

Worth separating two things that post #9 runs together.

Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.

I would put this at better than even and not much better.

Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.

Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.