of time. However, this investment is essential to maximising chances of success, and time invested at this stage will pay dividends later. Once the team and funding are in place and necessary approvals granted, the project can move on to developing the full protocol and beginning to collect, assemble and check the IPD that will be needed for analysis, as described in Chapter 4.
4 Running an IPD Meta‐Analysis Project: From Developing the Protocol to Preparing Data for Meta‐Analysis
Jayne F. Tierney, Richard D. Riley, Larysa H.M. Rydzewska, and Lesley A. Stewart
Summary Points
Collecting, checking and managing IPD is more involved and complex than extracting or collecting existing aggregate data, but is fundamental for ensuring that subsequent IPD meta‐analyses are robust and comprehensive.
An IPD meta‐analysis protocol should be registered or published, and include details of: the objectives, eligibility criteria, outcomes to be evaluated and the particular participant‐level covariates to be included or examined; the processes for collecting, coding and checking data and assessing risk of bias, and the statistical methods.
A separate comprehensive statistical analysis plan may also be necessary.
The project should aim to maximise the quantity of high‐quality IPD collected, in order to limit bias and uncertainty of results, and to complete the planned analyses reliably.
Negotiating and maintaining collaboration with trial investigators can take considerable time and effort, but is critical to the success of most IPD projects.
IPD should be transferred and stored securely, and used carefully and appropriately, in order to respect participant confidentiality, whilst also adhering to the data use agreements put in place with the data providers.
Variables contained in the supplied IPD will often need to be re‐coded, re‐defined or derived in a format that is common to all trials, to enable subsequent meta‐analysis.
Data checking is essential to help identify and rectify any major errors, inconsistencies or biases in the IPD, as well as promoting better understanding of individual trials. Such scrutiny may also afford credibility, which may be important for controversial or high‐profile projects.
A detailed log of all changes and transformations made to trial IPD should be maintained, to enable transparency and reproducibility.
Assessment of risk of bias for each trial is initially based on information from trial publications or protocols, supplemented by any information provided by the trial investigator, and is then refined once IPD have been received, checked and cleaned.
Some risk of bias concerns may be alleviated by having the IPD (e.g. inclusion of participants excluded from trial analyses), and others that may only become apparent from checking the IPD (e.g. an allocation pattern indicative of flawed randomisation).
Ultimately, all collaborators will have input into the interpretation and dissemination of results.
It can be helpful to incorporate patient and public involvement and engagement throughout the project, for example, to input into the development of the research questions, and the interpretation and dissemination of results.
4.1 Introduction
Once it has been established that an IPD meta‐analysis project is both appropriate (Chapter 2) and feasible (Chapter 3), and suitable personnel, governance and funding are in place (Chapter 3), there is a need to set up systems and processes that will ensure it will be conducted well. This chapter provides guidance on the preparations needed just prior to the collection of IPD (Section 4.2), and how to go about establishing and maintaining collaboration with IPD providers. It also outlines the steps (shown in Figure 4.1) needed to obtain, check and manage IPD (Sections 4.4 and 4.5), and to investigate risk of bias (Section 4.6), prior to final verification and merging of data (Sections 4.8 and 4.9) in readiness for analysis (Part 2). This stage is often underappreciated, but is critical to ensuring the success of the project and that subsequent meta‐analyses are robust and comprehensive. We focus on projects that aim to collate IPD from multiple randomised trials to examine the effect of a particular treatment, but very similar principles apply when seeking IPD from other study types, such as observational studies for diagnosis or prognosis (Part 5).
Figure 4.1 Overview of key steps involved in obtaining, managing and checking IPD from a single trial.
Source: Larysa Rydzewska and Jayne Tierney.
4.2 Preparing to Collect IPD
In this section, we discuss various aspects that should be considered when preparing to collect IPD. This critical set‐up stage occurs before formal requests for IPD are sent to trial investigators, but may still involve initial discussions with them, in order to gain clarity on the eligibility of their trials and the IPD they might be able to provide.
4.2.1 Defining the Objectives and Eligibility Criteria
Usually, the objectives and eligibility criteria for trials (and thus their IPD) will have already been defined, either in full or in part, using the PICOS or another similar framework,42 when developing a project scope (Section 3.3) or funding application (Section 3.9). However, they may still need to be refined at this stage. Figure 4.2 shows the PICOS elements that underlie the objective and eligibility criteria for an IPD meta‐analysis of pre‐operative chemotherapy for non‐small‐cell lung cancer.88
The type of trials deemed eligible for an IPD meta‐analysis project may depend not only on the specifics of the research question and the healthcare area, but also the proportion that are required for reliable synthesis, and are feasible to collect. Such elements should be explicitly considered in formulating the research question, and be built into the inclusion and exclusion criteria, with the rationale for doing so described. In healthcare areas where trials tend to be very large, it may not be worth including IPD from small trials if they would contribute little weight to the evidence base, but take a similar effort to obtain and manage as the larger