[2026 update] Confounding by indication, explained with a concrete example

post #5 is right about the mechanism and I think understates the practical bit.

Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

What makes a methodological criticism substantive: it identifies a specific feature of the design that materially affects what the paper can conclude. "Small sample size" alone is weak. "Small sample size for a rare outcome, so the confidence interval is wide" is stronger.

Picking up post #50: that is the part I would want checked first.

Bias towards the null and bias away from the null: different criticisms have different directions. Differential dropout might bias away from null; conservative statistical analysis might bias toward null.

Coming back to post #65, because the follow-up matters more than the original answer.

When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.

This follows post #68 rather than contradicting it.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

I read post #78 twice before replying, because I had assumed the opposite.

Two things before anyone answers the substance.

First, the context in the first post is clear and specific. Second, the question is framed so that an answer can actually address it. Both are the norm here and both matter more than they sound.

Coming back to post #96, because the follow-up matters more than the original answer.

Choosing the worst interpretation: "The confidence interval includes a harmful effect" is true if the CI goes from -1 to +5. But assuming the worst-case scenario is not how you use the evidence. The point estimate and the precision both matter.

Picking up post #100: that is the part I would want checked first.

I disagree with the reply above, and I think the disagreement is substantive rather than terminological.

The distinction being drawn does not survive when you look at the published data for this specific question. I would be glad to be shown wrong on this, because the version I am arguing against is more convenient.