according to the IPD meta‐analysis project protocol and SAP, this domain is not applicable.
Y: yes, PY: probably yes, N: no, PY: probably no, NI: no information. Note that in RoB 2, Domain 2 has an additional/alternative series of signalling questions for risk of bias arising from deviations from interventions in terms of adhering to interventions. The additional information that may be available from the IPD is similar to those given for the effect of assignment to the intervention in the table. Domain 5 has a sub‐list of questions that are not listed, because they are not relevant to IPD meta‐analysis projects.
Details of ways to evaluate the IPD in relation to the relevant RoB 2 domains are provided in the following sections. Findings of risk of bias and data checking can usefully be presented using a modified version of the ‘traffic light’ table used for standard risk of bias assessments (Section 4.7).
4.6.1 The Randomisation Process
For randomised trials, it is important to check that the methods of sequence generation and allocation concealment appear robust. This will help guard against the inclusion of non‐randomised trials, or non‐randomised participants, in the IPD meta‐analysis project. While a description of the methods can be gleaned from trial documentation and/or trial personnel, interrogating the IPD directly can highlight any unusual allocation patterns that may need further investigation.7,9,43
A simple method of checking the pattern of allocation is to look at the cumulative number of participants randomised into each treatment group over time. For example, Figure 4.7 shows the pattern for a trial included in an IPD meta‐analysis evaluating the effects of chemoradiation for cervical cancer.93 There are a similar number of participants allocated to each group throughout, and the curves cross frequently, which is what we would expect in a trial with robust randomisation procedures. It should be noted that for smaller trials, greater separation of curves and less crossing over might be expected, even if they are properly randomised, particularly if a simple method of randomisation was used. Also, where the allocation ratio is not 1:1, the curves would not be expected to cross, but rather would be expected to track one another.
Figure 4.7 The cumulative number of participants randomised to the intervention and control groups in a trial included in an IPD meta‐analysis of chemoradiation for cervical cancer.93
Source: Based on Chemoradiotherapy for Cervical Cancer Meta-Analysis Collaboration. Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials. Journal of clinical oncology: official journal of the American Society of Clinical Oncology 2008;26(35):5802–12.
A similar pattern is seen in the early stages of a trial included in an IPD meta‐analysis of chemotherapy versus radiotherapy for multiple myeloma, but for a short time period all the participants were allocated to the chemotherapy group (Figure 4.8).7 The trial investigator explained that this was when the radiotherapy machine had broken down, and all participants were given chemotherapy. As this particular cohort of participants were not randomly allocated to treatment, they were excluded from the IPD, thereby still allowing the trial to be included in the IPD meta‐analysis.
Figure 4.9 shows another example of how treatment allocation can be plotted, and was used to examine the pattern of assignment in a trial from another IPD meta‐analysis project. In this case, all participants except one were allocated to one intervention (arm 1) in the first months of the trial, and participants were allocated to the other intervention (arm 0) mostly in the final months. The trial investigator was unable to explain the pattern, agreed that it was not consistent with randomisation, and that the trial should be excluded from the meta‐analysis. As this was a single‐centre trial, where treatments were supplied in independently pre‐prepared trial packs (with no indication of the treatment they contained), it was speculated that these had not been mixed sufficiently.
Figure 4.8 The cumulative number of participants allocated to chemotherapy or radiotherapy in a trial included in an IPD meta‐analysis of treatments for multiple myeloma.
Source: Stewart et al.,7. © 1995, John Wiley & Sons.
Figure 4.9 Date (shown by year‐month) participants were allocated to treatment and control in a trial excluded from an IPD meta‐analysis, because participants in one group (‘arm 1’) were generally recruited earlier than those in the other group (‘arm 0’).
Source: Lesley Stewart.
As statistical software and database packages allow dates to be converted to days of the week, another simple check is to look at the number of participants allocated to the research and control groups on each day of the week.7,106 This method would highlight, for example, if participants were being allocated to treatment on particular clinic days (pseudo‐random allocation) or if participants were being allocated on the weekends, which would be unusual for trials in chronic conditions in many countries. For example, in a trial included in an IPD meta‐analysis examining pre‐operative chemotherapy for lung cancer,88 there were no weekend randomisations, and the numbers randomised to each treatment group were well balanced on the weekdays (Figure 4.10(a)). In contrast, for a trial included in an IPD meta‐analysis examining post‐operative radiotherapy for lung cancer,107 there appeared to be an unusually high number of weekend randomisations, and large imbalances in the number of participants allocated to each group on each day (Figure 4.10(b)). When this was brought to the attention of the trial investigator, they discovered problems with the management of the trial data, and went back to individual participant records, to ensure that the appropriate information was supplied, and the issues were resolved.
As mentioned in Section 4.4.1, if the dates of randomisation have been redacted from trial IPD for de‐identification purposes, it will not be feasible for the central research team to employ these checking procedures, but the trial statistician may be able to run these on their behalf. Moreover, it is still possible to visually check whether baseline characteristics appear reasonably balanced by group, as we would expect with a robust randomisation process. However, balance will never be perfect, and imbalances may be more pronounced in small trials or those with simple (i.e. non‐stratified) randomisation methods;