Distinguishing three things in the measurement error discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
Posting this under the heading it deserves: Measurement error in a self-reported exposure Everything below is what sits behind that.
The question about measurement error that I actually want answered is the second one below. The first is context and I have kept it short.
Both are stated with units, and I have said what I already checked so that nobody repeats it.
No disagreement from me. Posting only so the question does not look ignored.
The strongest argument against my own position on measurement error, stated as well as I can state it, since nobody else has yet.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
This is the sort of thing the wiki should carry and currently does not.
Distinguishing three things in the measurement error discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
That is clearer than the version I had in my head. Thank you.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Worth separating two things that post #6 runs together.
Measurement error is worth one more sentence than it usually gets, and the sentence is the one about how the number was arrived at.
Adding a small correction to the measurement error summary above rather than a disagreement with it. The substance holds; one of the figures is out by a factor that matters.
Worth separating two things that post #9 runs together.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
The literature is thinner on this than the confidence in the thread implies.
Post #13 is right about the mechanism and I think understates the practical bit.
What I want from this measurement error thread is the list of things that would need to be true for the claim to hold. If we can write that list, we can check it.
One caution on measurement error: everything above assumes the underlying documentation is what it claims to be. That assumption is doing real work and is rarely stated.
I read post #17 twice before replying, because I had assumed the opposite.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
This follows post #18 rather than contradicting it.
A request rather than an answer: could whoever has the primary source for measurement error post it? I have seen the claim three times this month and each version had lost a qualifier.
I read post #20 twice before replying, because I had assumed the opposite.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
The interesting part of this is the exception, and I do not understand the exception.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
It is one reading of the data and not the only reasonable one.
Picking up post #22: that is the part I would want checked first.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
If that reads as pedantic, it is, and it has saved me twice.
What would change my mind on measurement error is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.
The useful distinction on measurement error is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
This is the sort of exchange that makes the archive worth searching.