Pooling trials with different estimands — what changed since — setting out what I have, and where I think it stops being reliable.
What changes if the standard account of Pooling trials with different estimands is wrong? I ask because I have been treating it as settled and I noticed this week that I could not say why.
Working through the consequences rather than the evidence, since others here are better placed on the evidence.
Before the thread moves on from Pooling trials with different estimands — what is the sample size behind the claim? I am not being difficult; I have seen the same figure quoted from an n of four and from an n of four hundred.
That matches what I have seen, for whatever a single anecdote is worth.
Post #2 is right about the mechanism and I think understates the practical bit.
Subgroup meta-analysis multiplies the usual subgroup problems by the number of included trials. Treat it as hypothesis-generating without exception.
Posted with less confidence than the sentence structure implies.
For anyone finding this later: the short answer on Pooling trials with different estimands is that it depends on one thing, and the rest of the thread is people identifying which thing.
Picking up post #4: that is the part I would want checked first.
When trials differ in population, duration and comparator, the pooled estimate answers a question no individual trial asked. That is worth saying before the number is quoted.
I would want to see it done twice before believing it once.
On post #4 — agreed on the reasoning, with one qualification.
Practical note on Pooling trials with different estimands: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
Understood. Thank you for being specific about the limits of it.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
That is the version I would defend. It is not the version I started with.
Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.
A weak preference rather than a position.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
That matches what I was told, which is not the same as knowing it.
Post #12 is right about the mechanism and I think understates the practical bit.
My experience of Pooling trials with different estimands contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
Happy to be the one who is wrong here if it settles the question.
On post #15 — agreed on the reasoning, with one qualification.
Taking Pooling trials with different estimands seriously for a moment rather than deflecting: the honest position is that the community has observations and no controlled comparison, and those two things support very different sentences.
Following this. I have the same question and no better information than the first post.
The version of Pooling trials with different estimands that circulates here is a simplification of a simplification. It is not wrong, but it has lost the conditions under which it holds, and those conditions are where the interesting cases live.
I read post #20 twice before replying, because I had assumed the opposite.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
If that is already documented somewhere, ignore me and link it.
Sensible. I would want the same detail before I acted on it either.
Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.
The number is defensible. The precision I gave it is not.
Post #23 is right about the mechanism and I think understates the practical bit.
Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.
The strength of my opinion here exceeds the strength of my evidence.
Useful. I have added it to my own notes with the date on it.
Picking up post #23: that is the part I would want checked first.
A meta-analysis of four small trials is not stronger evidence than one adequately powered trial, whatever the summary statistic looks like.
Flagging that the sources on this are thinner than the confidence in the thread suggests.
Quality assessment of included trials should change the analysis rather than sit beside it. A sensitivity analysis excluding the weakest studies is the minimum.
Overlapping populations across included trials inflate the apparent sample size. It happens more than people expect where the same programme reports multiple papers.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
This follows post #27 rather than contradicting it.
When trials differ in population, duration and comparator, the pooled estimate answers a question no individual trial asked. That is worth saying before the number is quoted.
The literature is thinner on this than the confidence in the thread implies.