and 4.2.6, it may be helpful to undertake an initial risk of bias or quality assessment at the planning stages, before considering whether to obtain the IPD. However, the collection of IPD does allow a deeper and more reliable appraisal of data quality and risk of bias than is possible with aggregate data, because there is the opportunity to generate information directly from the IPD. There is also the potential to seek additional or updated trial IPD for inclusion in a meta‐analysis, in order to reduce or remove the potential for bias in particular domains. For example, participants excluded from original trial analyses may be reinstated in the meta‐analysis, or more appropriate statistical methods might be used. Therefore, an overall risk of bias assessment for each trial would be based on whether the design and conduct of the trial, and the quality of its final IPD (after correcting any data errors) are likely to lead to biased results when the IPD are analysed.
While assessing risk of bias in an IPD meta‐analysis involves using many of the same domains and items listed in the RoB 2 tool,91 some items are not relevant, because the IPD circumvents the issue. For example, the availability of IPD avoids reliance on a trial’s original analysis methods and reported results. Table 4.4 lists the domains and signalling questions in the RoB 2 tool that are particularly relevant to an IPD meta‐analysis project, and summarises how appraisal of the collected IPD, and other information harnessed through an IPD approach, can inform risk of bias judgements. These individual domains represent a useful starting point for assessing the risk of bias associated with each trial in an IPD meta‐analysis project.
Ultimately, if the trial design, management or conduct are seriously flawed (e.g. failed randomisation, poor follow‐up, or early stopping), this might lead to trials that are wholly or partially unreliable. Depending on the gravity of the issues, a trial may need to be excluded completely from the analyses or be the subject of sensitivity analysis; at the very least, the issues need to be brought to the attention of the reader in the IPD meta‐analysis publication. The IPD project’s central research team should have a plan in place for how they will deal with these sorts of occurrences. However, the objective of data checking is not to police trials or uncover fraud, and in our experience wholesale fabrication of trial data is uncommon. Also, note that this is probably less likely to occur in large, complex or multicentre trials, where collusion between multiple parties would be required. Nevertheless, sometimes trials are encountered where it is suspected that the data have been purposefully manipulated or falsified, so such trials would need to be excluded from the analyses, and the reasons for doing so described in the meta‐analysis report.
Table 4.4 Domains in the Risk of Bias 2 tool91 (RoB 2) of particular relevance to IPD meta‐analysis projects
Source: Based on Sterne JAC, Savovic J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898
| RoB 2 domains and signalling questions that are relevant to IPD meta‐analysis projects | Additional information available from IPD |
|---|---|
| Domain 1: Randomisation process | |
| 1.1 Was the allocation sequence random? 1.2 Was the allocation sequence concealed until participants were enrolled and assigned to interventions? 1.3 Did baseline differences between intervention groups suggest a problem with the randomisation process? | Assessing the pattern of randomisation using trial IPD may reveal instances where randomisation has failed, or may reassure that the randomisation process appears robust (Section 4.6.1). IPD can also be used to check balance across a full range of covariates (Section 4.6.1). Though formal testing is not recommended, visually, the distribution of each covariate by treatment group can help flag any systematic or unusual differences. |
| Domain 2: Deviations from the intended interventions (effect of assignment to intervention) | |
| 2.1 Were participants aware of their assigned intervention during the trial? 2.2 Were carers and people delivering the interventions aware of participants' assigned intervention during the trial? 2.3 If Y/PY/NI to 2.1 or 2.2: Were there deviations from the intended intervention that arose because of the trial context? 2.4 If Y/PY to 2.3: Were these deviations likely to have affected the outcome? 2.5 If Y/PY/NI to 2.4: Were these deviations from intended intervention balanced between groups? 2.6 Was an appropriate analysis used to estimate the effect of assignment to intervention? 2.7 If N/PN/NI to 2.6: Was there potential for a substantial impact (on the result) of the failure to analyse participants in the group to which they were randomised? | Having IPD means that participants can be analysed by their allocated intervention, irrespective of how they were analysed in the original trial (Section 4.6.2). Then, provided that patients, those delivering treatments or outcome assessors are not aware of the treatments being given, and the outcomes are objective, this would mean there is no risk of bias in this domain. If the IPD do not enable analysis by the allocated intervention, there may be sufficient detail in the trial dataset to check whether participants who deviated from intended intervention did so for pre‐specified or otherwise rational reasons. |
| Domain 3: Missing outcome data | |
| 3.1 Were data for this outcome available for all, or nearly all, participants randomised? 3.2 If N/PN/NI to 3.1: Is there evidence that the result was not biased by missing outcome data? 3.3 If N/PN to 3.2: Could missingness in the outcome depend on its true value? 3.4 If Y/PY/NI to 3.3: Is it likely that missingness in the outcome depended on its true value? | Access to IPD provides opportunity to request data for participants completely excluded from the IPD dataset, and for any outcomes missing for particular individuals (Section 4.6.3). Also, the IPD obtained can be examined to check for any potential undisclosed exclusions (Section 4.6.3). Where all the relevant IPD are provided, any data that were not analysed in the trial can be reinstated in the meta‐analysis, as appropriate. Also, methods for dealing with missing outcome data (e.g. mixed models) can be implemented, even if not undertaken by trial investigators in the original analysis. |
| Domain 4: Measurement of the outcome | |
| 4.1 Was the method of measuring the outcome inappropriate? 4.2 Could measurement or ascertainment of the outcome have differed between intervention groups? 4.3 If N/PN/NI to 4.1 and 4.2: Were outcome assessors aware of the intervention received by study participants? 4.4 If Y/PY/NI to 4.3: Could assessment of the outcome have been influenced by knowledge of intervention received? 4.5 If Y/PY/NI to 4.4: Is it likely that assessment of the outcome was influenced by knowledge of intervention received? | Access to IPD offers some opportunities to redress situations where inappropriate measurement may have been used. For example, component participant‐level data can be used to construct a new composite outcome (Section 4.6.4) and this may also reduce heterogeneity across trials. Even if a range of instruments have been used in trials, a more appropriate one may be obtained via the IPD. |
| Domain 5: Selection of the reported result | |
|
5.1 Were the data that produced this result analysed in accordance with a pre‐specified analysis plan that was finalized before unblinded outcome data were available for analysis?
| |