Pooling trials with different estimands

On the subject in the title: Pooling trials with different estimands Working notes rather than a conclusion.

Comparing SELECT (N Engl J Med, 2023) with SURMOUNT-1 (N Engl J Med, 2022) and finding the comparison harder than it looks.

Different populations, different durations, different endpoints defined slightly differently, and in one case a different estimand. People compare the headline percentages anyway, including me until recently.

Is there a defensible way to put these side by side, or is the honest answer that there is not and we should stop?

I read post #11 twice before replying, because I had assumed the opposite.

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.

Picking up post #38: that is the part I would want checked first.

Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.

Worth separating two things that post #51 runs together.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

post #68 is right about the mechanism and I think understates the practical bit.

Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

post #95 answers the question as asked. The question underneath it is different.

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.