This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Worth separating two things that post #27 runs together.
Practical note that does not fit anywhere else. Whatever you conclude from this topic, write down what you did and when. The single most useful thing in your own records is not any individual result; it is that they are dated and consecutive.
post #31 is right about the mechanism and I think understates the practical bit.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.
post #35 answers the question as asked. The question underneath it is different.
Bias towards the null and bias away from the null: different criticisms have different directions. Differential dropout might bias away from null; conservative statistical analysis might bias toward null.
Coming back to post #35, because the follow-up matters more than the original answer.
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Publication bias: a single published positive trial is weaker evidence than multiple published trials with consistent results. Asking whether there are unpublished negative trials is a fair critical question.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
Worth separating two things that post #38 runs together.
Practical note that does not fit anywhere else. Whatever you conclude from this topic, write down what you did and when. The single most useful thing in your own records is not any individual result; it is that they are dated and consecutive.
When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.
Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up.
Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.
Picking up post #44: that is the part I would want checked first.
Bias towards the null and bias away from the null: different criticisms have different directions. Differential dropout might bias away from null; conservative statistical analysis might bias toward null.
What makes a methodological criticism substantive: it identifies a specific feature of the design that materially affects what the paper can conclude. "Small sample size" alone is weak. "Small sample size for a rare outcome, so the confidence interval is wide" is stronger.
Publication bias: a single published positive trial is weaker evidence than multiple published trials with consistent results. Asking whether there are unpublished negative trials is a fair critical question.
Choosing the worst interpretation: "The confidence interval includes a harmful effect" is true if the CI goes from -1 to +5. But assuming the worst-case scenario is not how you use the evidence. The point estimate and the precision both matter.
On post #49 — agreed on the reasoning, with one qualification.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Choosing the worst interpretation: "The confidence interval includes a harmful effect" is true if the CI goes from -1 to +5. But assuming the worst-case scenario is not how you use the evidence. The point estimate and the precision both matter.
Worth separating two things that post #53 runs together.
When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.
post #57 is right about the mechanism and I think understates the practical bit.
For anyone arriving from a search: the marked solution above is the direct answer, and the replies underneath it add the caveats that make it safe to use.
Coming back to post #57, because the follow-up matters more than the original answer.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
Picking up post #57: that is the part I would want checked first.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.