22(10):1107-13.https://www.graceprinciples.org/publications.html
Early efforts on guidance documents produced checklists focused on quality reporting of observational research with items ranging from study background to bias control methods to funding sources (Table 1.2). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) was a collaboration of epidemiologists, journal editors, and other researchers involved in the conduct and reporting of observational research. The TREND group checklist was designed to mimic the CONSORT checklist for randomized controlled trials. Both of these efforts produced 22-item checklists and reminded those disclosing observational research of the core issues that were both common to randomized research reporting and the unique reporting issues for observational research.
The next set of guidance documents was largely led by key professional societies involved in the conduct and reporting of real world evidence. The Good Research for Comparative Effectiveness (GRACE) principles was a collaboration between experienced academic and private researchers and the International Society of Pharmacoepidemiology (ISPE). This began with a set of quality principles published in 2010 that could be used to assess the quality of comparative observational research and provided a set of good practice principles regarding the design conduct, analysis, and reporting of observational research. These principles were further developed into a checklist, which was validated as a tool through multiple research studies.
The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) commissioned a task force to develop its own guidance with a goal of providing more detail than a checklist as well as covering more of the research process. Specifically, they began with guidance on developing the research question and concluded with much more detail regarding methods for control of confounding. The end result was a three-paper series concluding with a focused discussion of analytic methods.
More recently, joint efforts have produced further quality guidance for researchers developing and disclosing observational studies. A joint ISPOR-ISPE task force was created to produce good procedural practices that would increase decision maker’s confidence in real world evidence. The intent here was to build on the earlier separate work from ISPE and ISPOR on the basic principles and address the transparency of observational research. Specifically, this covered seven topics including study registration, replicability, and stakeholder involvement. For instance, these guidelines recommend a priori registration of hypothesis evaluating treatment effectiveness (HETE) studies for greater credibility.
ISPOR, the Academy of Managed Care Pharmacy (AMPC), and the National Pharmaceutical Council (NPC) jointly produced a document to guide reviewers on the degree of confidence one can place on a specific piece of observational research as well as further educate the field on the subtleties of observational research issues. The format used was a questionnaire in flowchart format that focused on issues of credibility and relevance.
Recently, the debate has focused on the potential regulatory use of RWE. This has been hastened by the 21st Century Cures Act, which mandates the FDA to produce a guidance document regarding regulatory decision making with RWE. The FDA had previously released guidance for industry on the use of RWE for regulatory decision making for medical devices. A main focus of this document was on ensuring the quality of the data – as much real world data is not captured in a research setting and inaccurate recordings of diagnoses and outcome ascertainment can seriously bias analyses. The Duke-Margolis Center for Health Policy has taken up leadership in the debate on regulatory use of RWE and organized multiple stakeholders to develop a framework for the regulatory use of RWE. They released a white paper (Duke Margolis Center for Health Policy, 2017) that discusses what quality steps are necessary for the development and conduct of real world evidence that could be fit for regulatory purposes. Most recently (December 2018), the FDA released a framework for the use of RWE for regulatory decision making. This outlines how the FDA will evaluate the potential use of RWE to support new indications for approved drugs or satisfy post-approval commitments.
Also of note is the Get Real Innovative Medicine Initiative (IMI), a European consortium of pharmaceutical companies, academia, HTA agencies, and regulators. The goals are to speed the development and adoption of new RWE-related methods into the drug development process. A series of reports or publications on topic such as assessing the validity or RWE designs and analysis methods and innovative approaches to generalizability have been or are under development (http://www.imi-getreal.eu).
Common themes among all of the guidance documents include pre-specification of analysis plans, ensuring appropriate and valid outcome measurement (data source), adjustment for biases, and transparency in reporting.
1.8 Best Practices for Real World Research
Regarding the process for performing a comparative analysis from real world data, we follow the proposals of Rubin (2007) and Bind and Rubin (2017), which are in alignment with the guidance documents in Table 1.2. Specifically, they propose four stages for a research project:
1. Conceptual
2. Design
3. Statistical Analysis
4. Conclusions
In the initial conceptual stage, researchers conceptualize how they would conduct the experiment as a randomized controlled trial. This allows the development of a clear and specific causal question. At this stage we also recommend following the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) E9 guidance of carefully defining your estimand after the objectives of the study are developed. The estimand consists of the population that you want to draw inference to, the outcome to be measured on each patient, intercurrent events (for example, post initiation events such as switching of medications, non-adherence), and