Research Methods · Statistics

Multiplicity when you track fifteen variables

Research Methods › Statistics effect size heterogeneity confounding worked example

Solved

Solved by n.achebe in post #3

Absence of evidence and evidence of absence: if a study is small and finds no effect, that is absence of evidence, not evidence of absence. A larger study might find an effect that a small study missed.

Jump to the accepted answer →

j.asanteTL2 Moderator17 Jun 2025#1

Multiplicity when you track fifteen variables — setting out what I have, and where I think it stops being reliable.

Comparing LEADER (N Engl J Med, 2016) with SURMOUNT-OSA (N Engl J Med, 2024) and finding the comparison harder than it looks.

Different populations, different durations, different endpoints defined slightly differently, and in one case a different estimand. People compare the headline percentages anyway, including me until recently.

Is there a defensible way to put these side by side, or is the honest answer that there is not and we should stop?

1 like 13mo

t.nardoneTL3Regular24 Jul 2025#2

I disagree with the reply above, and I think the disagreement is substantive rather than terminological.

The distinction being drawn does not survive when you look at the published data for this specific question. I would be glad to be shown wrong on this, because the version I am arguing against is more convenient.

0 likes 12mo

n.achebeTL2 Moderator Solution20 Aug 2025#3

Absence of evidence and evidence of absence: if a study is small and finds no effect, that is absence of evidence, not evidence of absence. A larger study might find an effect that a small study missed.

20 likes 11mo

This topic was referenced in

Regression to the mean in progress reportsResearch Methods › Statistics · 34 replies
Measurement error in home scales, with a worked standard deviation — the long versionResearch Methods › Statistics · 105 replies

Suggested topics

Topic	Participants	Replies	Views	Activity
Correlation in a self-tracked dataset: what it can support Research Methods › Statistics effect size heterogeneity worked example number needed to treat Closed Correlation in a self-tracked dataset: what it can support — setting out what I have, and where I think it stops being reliable. I have seen STEP 4 ( JAMA , 2021) cited in support of a claim I do not think it…	+152	166	48k	23mo
Sample size intuition for a personal experiment — does this still hold? Research Methods › Statistics effect size heterogeneity number needed to treat Solved Sample size intuition for a personal experiment — does this still hold? I have a specific reason for asking rather than idle curiosity, and the context is below. Comparing STEP 1 ( N Engl J Med , 2021) with…	+33	38	26k	23d
Second pass at: Absence of evidence and evidence of absence Research Methods › Statistics worked example confounding number needed to treat On the subject in the title: Second pass at: Absence of evidence and evidence of absence Working notes rather than a conclusion. Session topic: SELECT ( N Engl J Med , 2023). Please read it before posting;…	+50	55	18k	16h
Second pass at: Multiplicity when you track fifteen variables Research Methods › Statistics number needed to treat confounding effect size worked example Posting this under the heading it deserves: Second pass at: Multiplicity when you track fifteen variables Everything below is what sits behind that. I have seen SCALE ( N Engl J Med , 2015) cited in support…		2	12k	18mo
Absence of evidence and evidence of absence Research Methods › Statistics effect size worked example number needed to treat Absence of evidence and evidence of absence Writing it up because I had to work it out twice and would rather nobody else did. Session topic: SURMOUNT-OSA ( N Engl J Med , 2024). Please read it before…	+16	20	3.9k	5mo

Related topics — sharing the tags effect size, heterogeneity, confounding

Topic	Participants	Replies	Views	Activity
Second pass at: Why semaglutide's albumin binding is the whole reason weekly dosing works Compounds › Semaglutide SELECT trial STEP programme effect size randomised trial Second pass at: Why semaglutide's albumin binding is the whole reason weekly dosing works Writing it up because I had to work it out twice and would rather nobody else did. I have seen SURPASS-4 ( Lancet ,…		2	39k	6mo
Why we ask for units every single time, with three examples of what goes wrong — what changed since Start Here › How to Ask onboarding guidelines worked example beginner friendly The question in the title: Why we ask for units every single time, with three examples of what goes wrong — what changed since I will give what I have already checked below so nobody repeats it. Question in…	+14	18	19k	2mo
Revisiting: Holding a dose indefinitely: is there a documented downside? Practice › Dosing & titration titration tirzepatide worked example missed dose Solved The question in the title: Revisiting: Holding a dose indefinitely: is there a documented downside? I will give what I have already checked below so nobody repeats it. A question about technique rather than…	+26	30	4.5k	18mo
Washout with a one-week half-life: the arithmetic Research Methods › N-of-1 designs effect size data table worked example Wiki Washout with a one-week half-life: the arithmetic — setting out what I have, and where I think it stops being reliable. I have seen LEADER ( N Engl J Med , 2016) cited in support of a claim I do not think it…	+90	96	44k	20mo
Feature request: should a dose-equivalence tool exist at all? Data & Tools › Calculators site feedback worked example concentration maths syringe units Closed Feature request: should a dose-equivalence tool exist at all? — that is the question, and I have not found it answered plainly anywhere I have looked. Posting the method first, because I know what the first…	+21	25	4.8k	18mo

Sponsored Sigma-Aldrich — certified reference materialsReference standards, analytical reagents and certified materials for method calibration. If you are validating a method, this is where the standard comes from.Reference standards →