Heterogeneity as information rather than as a nuisance

Posting this under the heading it deserves: Heterogeneity as information rather than as a nuisance Everything below is what sits behind that.

Session topic: SCALE (N Engl J Med, 2015). Please read it before posting; the discussion is much better when everyone has.

The question I would like us to start with is what the trial set out to estimate, rather than what it found. Once that is on the table we can talk about whether the design could have answered it, and only then about the numbers.

Specific things I would like covered: the population and how far it generalises, how discontinuation was handled, whether the comparator was a fair one, and what the absolute rather than relative effect looks like.

I will summarise at the end and the summary will feed the relevant digest page.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

This follows post #2 rather than contradicting it.

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.

I read the opening post twice before replying, because I had assumed the opposite.

Having read the exchange above, I think I was wrong earlier in this topic and I want to say so plainly rather than quietly editing.

The correction was fair and I had been repeating something I had not checked carefully enough.

Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.

On post #2 — agreed on the reasoning, with one qualification.

Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.

Picking up post #4: that is the part I would want checked first.

Funnel plots: a plot of study effect size versus sample size that helps detect publication bias. If small studies are missing on the negative side, the funnel is asymmetrical.

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

post #8 is right about the mechanism and I think understates the practical bit.

Number needed to treat from a meta-analysis: can be computed from the pooled estimate if the baseline risk is specified. More interpretable than pooled relative effects.

Two things before anyone answers the substance.

First, the context in the first post is clear and specific. Second, the question is framed so that an answer can actually address it. Both are the norm here and both matter more than they sound.

Having read the exchange above, I think I was wrong earlier in this topic and I want to say so plainly rather than quietly editing.

The correction was fair and I had been repeating something I had not checked carefully enough.

Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.

Worth separating two things that post #9 runs together.

When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.

post #13 is right about the mechanism and I think understates the practical bit.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.