can occur if certain participants are followed‐up for a longer duration than others, such that event rates appear higher. Therefore, for these outcomes, it is also desirable to check that follow‐up is balanced by treatment group. This can be achieved by selecting those trial participants who are event‐free, then using censoring as the event and the date of censoring as the time‐to‐event in a ‘reverse Kaplan‐Meier’ analysis. In one trial included in an IPD meta‐analysis of adjuvant chemotherapy for soft tissue sarcoma,102 all participants who were still alive had been followed for death for a minimum of about nine years, and subsequently to the same degree in both treatment groups (Figure 4.12(a)). In another smaller trial from the same meta‐analysis, although participants were followed for a long time, there was an imbalance by group (Figure 4.12(b)), which became less of an issue when the trial investigator provided updated IPD with extended follow‐up (Figure 4.12(c)).
Trials that stop early can produce results that are overly in favour of treatment or control,117 and therefore introduce bias into a subsequent meta‐analysis. For those based on time‐to‐event outcomes, obtaining IPD with updated follow‐up can go some way to addressing this issue. For example, in an IPD meta‐analysis examining adjuvant chemotherapy for locally advanced bladder cancer, three of the included trials were stopped early, because they had interim results in favour of adjuvant treatment. However, the IPD meta‐analysis project helped alleviate this potential bias, as it included IPD with updated follow‐up for the three trials, which produced results that were less in favour of adjuvant treatment (Table 4.5).118 IPD also allows non‐proportional hazards (non‐constant hazard ratios) to be examined (see Chapter 5).
Figure 4.12 ‘Reverse’ Kaplan‐Meier analysis of participants who are event‐free for (a) a trial with balanced follow‐up, and (b) a trial with imbalanced follow‐up that was (c) subsequently updated with longer follow‐up when IPD were collected. Each was included in an IPD meta‐analysis of adjuvant chemotherapy for soft tissue sarcoma.102
Source: Sarcoma Meta-Analysis Collaboration. Adjuvant chemotherapy for localised resectable soft-tissue sarcoma of adults: meta-analysis of individual data. Lancet 1997;350(9092):1647–54.
Table 4.5 Alleviating potential bias in trials that stopped early for perceived benefit (included in an IPD meta‐analysis of adjuvant chemotherapy for locally advanced bladder cancer) through updated follow‐up
Source: Jayne Tierney, adapted with permission.118
| Trial | Skinner | Studer | Stockle |
|---|---|---|---|
| Outcome analysed | Survival | Survival | Disease‐free survival |
| % participants with updated follow‐up since published analysis | 100 | 22 | 100 |
| Hazard ratio estimated from published statistics or Kaplan‐Meier curves | 0.65 | 0.86 | 0.39 |
| Hazard ratio derived from IPD | 0.75 | 1.02 | 0.45 |
4.7 Assessing and Presenting the Overall Quality of a Trial
The results of validity checking (Section 4.5) and risk of bias assessment (Section 4.6) should be considered together in order to build up an overall picture of the quality of each trial’s IPD. This should include reflections on the quality of the trial design and conduct (from the ROB 2 assessment), checks of IPD obtained, and any unresolved errors or concerns therein. If it is concluded that the IPD from a particular trial is likely to introduce considerable bias into an IPD meta‐analysis, then it is may be sensible to exclude it. For example, in an IPD meta‐analysis of post‐operative therapy for non‐small‐cell lung cancer,107 a trial was excluded because it ‘failed’ the data checks,101 and it is certainly worth highlighting any such exclusions in the relevant meta‐analysis publication. However, such situations need to be handled sensitively with trial investigators, who will have invested time and effort in supplying the data, and may have been unaware that issues would emerge. Alternatively, the impact of risk of bias may be explored through sensitivity analysis, such as examining how meta‐analysis conclusions change according to whether or not trials have risk of bias concerns (Chapter 9).
The ‘traffic light’ table used for standard risk of bias assessment can be usefully adapted for summarising the overall quality of trials included in an IPD meta‐analysis. This would include all the domains described previously, except the “selection of the reported results” domain, which is not applicable to IPD projects (Section 4.6), because the trial IPD are re‐analysed according to the meta‐analysis protocol and SAP. Adopting this structure means that the information is readily comparable with a standard risk of bias table, but can easily be extended to incorporate columns for additional project‐specific IPD checks that are deemed particularly important. For example, it is useful to include an additional column to indicate whether there were any residual concerns about data quality once the data checking and correction procedures had been completed, and which might impact on the trustworthiness of a trial. Often these risk of bias ‘traffic light’ tables will be almost completely green (low risk of bias), as any trials that fail data checking or have serious bias issues would likely be excluded from the IPD meta‐analysis project completely. Note that assessments should be based on the fullest information possible, thereby considering the trial design and conduct based on all trial documentation and contact with investigators, plus the results of checks of the IPD. A more detailed risk of bias table might be included in an appendix to provide fuller information on this data checking process (Table 4.6), and show how individual judgements have been arrived at. It also provides an opportunity to flag less serious or unclear bias issues.
Table 4.6 Excerpt of a RoB2 table for an IPD meta‐analysis of adjuvant chemotherapy for locally advanced bladder cancer based on a single trial and the main outcome of overall survival (Tierney et al., in preparation).
Source: Sarah Burdett and Jayne Tierney.
| Risk of Bias Domain | 1) Randomisation process |
2) Deviations from the intended
|
|---|