Evidence · Study critique

About the Study critique category

Evidence › Study critique risk of bias confounding observational data

Pinned Wiki

bench_notesTL4 Moderator12 Nov 2024#1

Community wiki post. Any member at trust level 3 or above can edit this post; every edit is recorded. Last edited by hana.sato on 5 Jan 2025.

17 Jan 2025 — bench_notes: Added the worked example and a unit label to the table header.
1 May 2025 — mira.patel: Clarified the distinction that was causing repeat questions below.
5 Jan 2025 — hana.sato: Removed a claim that the cited source did not support.

Editors: bench_notes, mira.patel, hana.sato

Staff post. Actions described here are recorded in the public moderation log and may be challenged in Meta.

Structured criticism of a specific published study. Criticise the method, not the author.

This post is a community wiki: any member at trust level 3 or above can edit it, and every edit is recorded with its author and a note. If the scope described here has drifted from what actually gets posted, edit it rather than starting a topic about it.

What belongs here

Questions specific to study critique, with the units and the context stated.
Reports and datasets, with the collection method stated.
Disagreement about the substance, conducted in public.

What belongs elsewhere

Anything urgent belongs in Safety, immediately, and not here.
Anything about how this site works belongs in Meta.
Requests for a maintained page belong in Wiki requests.

Before posting, it is worth checking the documentation commons and this category's tag pages. A great deal of what gets asked here has a maintained answer with a review date on it, which is more reliable than a two-year-old reply.

0 likes 20mo

bench_notesTL4 Moderator29 Dec 2024#2

Staff post. Actions described here are recorded in the public moderation log and may be challenged in Meta.

Moderation notes for this category, so that nothing about how it is run is a surprise.

Topics get moved into and out of here regularly. A move is not a judgement — the category tree is not obvious from outside and putting a question in the wrong place is expected. When I move something I say so in the topic and the action appears in the public log with the guideline cited.

The two guidelines that come up most often in this category are R2, on supplying a source when asked, and R7, on category placement. Neither is used punitively.

3 likes 19mo

j.nwosuTL2 Moderator1 Feb 2025#3

As a member rather than staff: the most useful thing I did when I started reading this category was to work through the tag pages rather than the topic list. The topic list shows you what is recent. The tags show you what is settled.

10 likes 18mo

d.eriksenTL2 Moderator3 Mar 2025#4

Suggestion for the wiki post above: it would be worth linking the two or three most-referenced topics in this category directly, since new members reliably ask for exactly those. Happy to make the edit myself if nobody objects, with a revision note.

22 likes 17mo

h.falkTL2 Moderator30 Mar 2025#5

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

0 likes 16mo

This topic was referenced in

[2026 update] Confounding by indication, explained with a concrete exampleEvidence › Study critique · 116 replies

Suggested topics

Topic	Participants	Replies	Views	Activity
Immortal time bias in a claims-database study — one year on Evidence › Study critique effect size risk of bias confounding Solved Posting this under the heading it deserves: Immortal time bias in a claims-database study — one year on Everything below is what sits behind that. Comparing SURPASS-2 ( N Engl J Med , 2021) with SURMOUNT-1 (…	+41	46	3.7k	22h
Critiquing an observational claim about a class effect — one year on Evidence › Study critique observational data disputed risk of bias Pinned Critiquing an observational claim about a class effect — one year on — setting out what I have, and where I think it stops being reliable. I have seen LEADER ( N Engl J Med , 2016) cited in support of a claim…	+5	9	3.2k	8mo
A structured critique template this community uses — a second dataset Evidence › Study critique observational data effect size confounding Solved On the subject in the title: A structured critique template this community uses — a second dataset Working notes rather than a conclusion. Session topic: SELECT ( N Engl J Med , 2023). Please read it before…	+47	51	16k	15h
Critiquing an observational claim about a class effect Evidence › Study critique disputed risk of bias effect size observational data Posting this under the heading it deserves: Critiquing an observational claim about a class effect Everything below is what sits behind that. Session topic: SCALE ( N Engl J Med , 2015). Please read it before…		2	6.9k	14mo
A structured critique template this community uses Evidence › Study critique observational data effect size risk of bias A structured critique template this community uses — setting out what I have, and where I think it stops being reliable. Comparing SURMOUNT-2 ( Lancet , 2023) with SURPASS-4 ( Lancet , 2021) and finding the…		2	18k	1d

Related topics — sharing the tags risk of bias, confounding, observational data

Topic	Participants	Replies	Views	Activity
TB-500 and thymosin beta-4: the fragment versus the protein Compounds › Repair & healing peptides observational data TB-500 BPC-157 preprint TB-500 and thymosin beta-4: the fragment versus the protein Writing it up because I had to work it out twice and would rather nobody else did. I have seen SURPASS-4 ( Lancet , 2021) cited in support of a…	+107	118	22k	11mo
Journal club: indirect comparison between two programmes, defended and attacked Evidence › Journal club journal club randomised trial number needed to treat risk of bias Journal club: indirect comparison between two programmes, defended and attacked Writing it up because I had to work it out twice and would rather nobody else did. Comparing SUSTAIN 6 ( N Engl J Med , 2016)…	+122	131	5.2k	just now
About the Repair & healing peptides category Compounds › Repair & healing peptides BPC-157 TB-500 observational data Pinned Wiki BPC-157, TB-500 and similar. Mostly preclinical evidence; this subcategory is strict about not overstating it. This post is a community wiki: any member at trust level 3 or above can edit it, and every edit…		4	19k	13mo
Journal club: STEP 4 and what a withdrawal design can prove — one year on Evidence › Journal club number needed to treat estimand randomised trial effect size Closed Journal club: STEP 4 and what a withdrawal design can prove — one year on Writing it up because I had to work it out twice and would rather nobody else did. Session topic: SURMOUNT-OSA ( N Engl J Med , 2024).…	+56	62	34k	15mo
Pooling trials with different estimands — what changed since Evidence › Meta-analyses publication bias risk of bias meta-analysis effect size Pooling trials with different estimands — what changed since — setting out what I have, and where I think it stops being reliable. Comparing STEP 4 ( JAMA , 2021) with SURPASS-4 ( Lancet , 2021) and finding…	+57	64	12k	7mo

Sponsored Sigma-Aldrich — certified reference materialsReference standards, analytical reagents and certified materials for method calibration. If you are validating a method, this is where the standard comes from.Reference standards →