What pooling buys you and what it destroys

The question in the title: What pooling buys you and what it destroys I will give what I have already checked below so nobody repeats it.

Comparing SURMOUNT-2 (Lancet, 2023) with LEADER (N Engl J Med, 2016) and finding the comparison harder than it looks.

Different populations, different durations, different endpoints defined slightly differently, and in one case a different estimand. People compare the headline percentages anyway, including me until recently.

Is there a defensible way to put these side by side, or is the honest answer that there is not and we should stop?

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

Coming back to post #23, because the follow-up matters more than the original answer.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

This follows post #34 rather than contradicting it.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

Worth separating two things that post #41 runs together.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

Picking up post #67: that is the part I would want checked first.

Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.

Picking up post #74: that is the part I would want checked first.

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.

post #81 is right about the mechanism and I think understates the practical bit.

When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.