The Peptide CommonsEst. May 2024
Search
Independent. We sell nothing and are affiliated with no manufacturer or pharmacy. Every moderation action is logged in public
Evidence · Meta-analyses

Pooling trials with different estimands

Evidence › Meta-analysespublication bias risk of bias effect size meta-analysis
Solved
Solved by taper_shift in post #3
Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up.

Jump to the accepted answer →

RC
r.coelhoTL2 Moderator30 May 2026#1

On the subject in the title: Pooling trials with different estimands Working notes rather than a conclusion.

Comparing SELECT (N Engl J Med, 2023) with SURMOUNT-1 (N Engl J Med, 2022) and finding the comparison harder than it looks.

Different populations, different durations, different endpoints defined slightly differently, and in one case a different estimand. People compare the headline percentages anyway, including me until recently.

Is there a defensible way to put these side by side, or is the honest answer that there is not and we should stop?

34 likes 2mo
FC
f.chowdhuryTL2 Moderator31 May 2026#2

Funnel plots: a plot of study effect size versus sample size that helps detect publication bias. If small studies are missing on the negative side, the funnel is asymmetrical.

0 likes 2mo
TS
taper_shiftTL3Regular Solution1 Jun 2026#3

Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up.

7 likes 2mo
RM
r.molnarTL2 Moderator2 Jun 2026#4
taper_shift, post #3: Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up. Go to post

Worth separating two things that post #3 runs together.

Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.

4 likes in reply to #3 2mo
ME
m.eriksenTL2 Moderator3 Jun 2026#5

Picking up post #2: that is the part I would want checked first.

Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.

25 likes 2mo
ME
m.ekstromTL2 Moderator4 Jun 2026#6

Coming back to post #4, because the follow-up matters more than the original answer.

When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.

0 likes 2mo
T
ThibodeauTL3Regular4 Jun 2026#7

Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.

1 like 2mo
MR
m.restrepoTL2 Moderator5 Jun 2026 · edited#8
m.ekstrom, post #6: Coming back to post #4, because the follow-up matters more than the original answer. When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the… Go to post

Number needed to treat from a meta-analysis: can be computed from the pooled estimate if the baseline risk is specified. More interpretable than pooled relative effects.

7 likes in reply to #6 2mo
LS
l.solbergTL2 Moderator6 Jun 2026 · edited#9
r.molnar, post #4: Worth separating two things that post #3 runs together. Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies. Go to post

I disagree with the reply above, and I think the disagreement is substantive rather than terminological.

The distinction being drawn does not survive when you look at the published data for this specific question. I would be glad to be shown wrong on this, because the version I am arguing against is more convenient.

8 likes in reply to #4 2mo
EV
e.verhoevenTL2 Moderator6 Jun 2026#10

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

19 likes 2mo
IL
integrator_logTL3Regular7 Jun 2026#11

Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.

5 likes 2mo
GO
g.oyelaranTL2 Moderator8 Jun 2026#12

post #11 is right about the mechanism and I think understates the practical bit.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

0 likes 2mo
VT
vial_tableTL2Member8 Jun 2026#13
r.molnar, post #4: Worth separating two things that post #3 runs together. Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies. Go to post

I read post #11 twice before replying, because I had assumed the opposite.

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.

30 likes in reply to #4 2mo
SS
s.salgadoTL2 Moderator9 Jun 2026 · edited#14

Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up.

15 likes 2mo
F
FairweatherTL2Member9 Jun 2026#15

On post #11 — agreed on the reasoning, with one qualification.

Number needed to treat from a meta-analysis: can be computed from the pooled estimate if the baseline risk is specified. More interpretable than pooled relative effects.

3 likes 2mo
DV
d.vestergaardTL2 Moderator10 Jun 2026#16

post #15 answers the question as asked. The question underneath it is different.

Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis.

0 likes 2mo
SF
sterile_fileTL3Regular10 Jun 2026#17
Thibodeau, post #7: Subgroup analysis: sometimes a meta-analysis reports separate pooled estimates for different subgroups (e.g., by baseline body mass index or by trial duration). Be cautious — many subgroup analyses are exploratory and less reliable than the main analysis. Go to post

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

22 likes in reply to #7 2mo
KB
ka.batistaTL2 Moderator11 Jun 2026#18
r.molnar, post #4: Worth separating two things that post #3 runs together. Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies. Go to post

Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.

10 likes in reply to #4 2mo
EP
e.piresTL2 Moderator11 Jun 2026#19

Worth separating two things that post #15 runs together.

Publication bias: what did not get published? Small studies with negative results are less likely to be published than large studies with positive results. A forest plot with only large studies on the positive end is a red flag for unpublished small negative studies.

1 like 2mo
AK
a.kowalskiTL2 Moderator12 Jun 2026#20

I disagree with the reply above, and I think the disagreement is substantive rather than terminological.

The distinction being drawn does not survive when you look at the published data for this specific question. I would be glad to be shown wrong on this, because the version I am arguing against is more convenient.

0 likes 2mo
B
BirkelandTL3Regular12 Jun 2026#21
g.oyelaran, post #12: post #11 is right about the mechanism and I think understates the practical bit. Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question. Go to post

Why forest plots are more informative than pooled numbers: they show the variation across studies, which tells you whether the effect is consistent or heterogeneous. A narrow confidence interval around a meaningless centre is less useful than a wider interval that shows real differences.

2 likes in reply to #12 1mo
PF
p.fontaineTL2 Moderator13 Jun 2026#22

Funnel plots: a plot of study effect size versus sample size that helps detect publication bias. If small studies are missing on the negative side, the funnel is asymmetrical.

9 likes 1mo
NR
n.rowntreeTL3Regular13 Jun 2026#23

Picking up post #20: that is the part I would want checked first.

Pooled estimates and heterogeneity: when trials differ in population, duration, or comparator, a pooled estimate answers a question that no individual trial asked. High heterogeneity means effects genuinely differ across studies. The pooled number is an average of things that should not have been averaged.

20 likes 1mo
MO
m.oyelaranTL2 Moderator14 Jun 2026#24
m.ekstrom, post #6: Coming back to post #4, because the follow-up matters more than the original answer. When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the… Go to post

Coming back to post #22, because the follow-up matters more than the original answer.

When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.

0 likes in reply to #6 1mo
FE
footnote_entryTL3Regular14 Jun 2026#25
g.oyelaran, post #12: post #11 is right about the mechanism and I think understates the practical bit. Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question. Go to post

post #24 is right about the mechanism and I think understates the practical bit.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

5 likes in reply to #12 1mo
HC
h.castellanosTL2 Moderator15 Jun 2026 · edited#26

Having read the exchange above, I think I was wrong earlier in this topic and I want to say so plainly rather than quietly editing.

The correction was fair and I had been repeating something I had not checked carefully enough.

13 likes 1mo
K
KStephanopoulosTL3Regular15 Jun 2026#27

Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.

28 likes 1mo
SV
s.vogelTL2 Moderator16 Jun 2026#28

I read post #26 twice before replying, because I had assumed the opposite.

Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.

0 likes 1mo
I
IsaksenTL3Regular16 Jun 2026#29
integrator_log, post #11: Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high. Go to post

Thank you for the correction. I have edited my earlier post with a note rather than silently, so the thread still makes sense to read. The error was mine and it was the kind that comes from remembering a figure instead of looking it up.

0 likes in reply to #11 1mo
RC
r.coelhoTL2 Moderator17 Jun 2026#30
ka.batista, post #18: Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded. Go to post

On post #26 — agreed on the reasoning, with one qualification.

Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.

2 likes in reply to #18 1mo
Sponsored Sigma-Aldrich — certified reference materialsReference standards, analytical reagents and certified materials for method calibration. If you are validating a method, this is where the standard comes from.Reference standards →