Real-World Evidence Navigator

A first of its kind RWE educational resource providing guidance and a directory of resources

Sources of Real-world Data
Generating Rwe
Addressing Bias
Research Governance
Guidance Documents

Introduction

With the increased availability of data, developments in analytical approaches and study designs and the emergence of adaptive pathways for the licensing and reimbursement, there has been growing interest in the use of real-world data in the development of new medical technologies.

Using the RWE Navigator will help to understand the application of RWE in the development of medicines, vaccines and technologies, including data sources, study designs and analytical techniques with reference to authoritative sources if you want to find out more.

The Real-World Evidence (RWE) Navigator:

Is An Educational Resource

Helping users to find out more about using real-world data to understand the diseases and the value of medical
treatments.

Provides Guidance

Guiding users to specific types of analyses or study designs using RWE to support the development of medical technologies.

Is A Directory of Resources

A comprehensive resource on the use of RWE in evaluation of medical technologies, signposting to outputs from authoritative sources of information on RWE.

What is Real-World Data?

Real-world data (RWD) is typically defined as data relating to patient health or experience or care delivery collected outside the context of a highly controlled clinical trial.
Precise definitions of real-world data vary across institutions. All consider observational data from retrospective or prospective designs as real-world data. Some additionally include data collected in pragmatic clinical trials.
Is A DirectorCommon types of real-world data include patient demographics, healthcare utilisation, clinical outcomes, medical history, test results, imaging, and
free text.
Sources of Real World Data

Overview

Real-world data sources include electronic health records, patient registries, administrative or claims data, prospective cohorts, surveys, pragmatic clinical
trials, or patient-generated data (see Table 1). These sources are not always independent. 

  • Patient registries can be formed from electronic health data.
  • Disparate data sources may be linked. For instance, in the UK, electronic health data from primary care is often linked to hospital administrative

For instance, Table 1 provides a high-level overview of key sources of real-world data. Use the links to see more information on each source

Table 1 — Common Sources of Real-World Data
Data SourceSummary

Electronic Health Records		Systems into which healthcare providers enter routine clinical data during usual practice. These sometimes integrate data from other information systems, including laboratory, genomic, and imaging systems. These are typically used to inform the clinical management of patients.
Patient RegistriesPatient registries are organised systems that collect uniform data to identify specified outcomes for a population defined by a particular disease, condition or exposure (AHRQ 2020). Registries can serve several purposes including research, clinical care or policy.
Administrative or Claims DataData collected for administrative purposes including healthcare planning and financing. Claims data are administrative data on the use of healthcare services, often collected in insurance-based systems.
Observational Cohorts With Primary Data CollectionTraditional prospective studies that are designed to answer one or more research questions.
Health SurveysInvolve the systematic collection of data about health and disease in a human population through survey methods. They have various purposes, including understanding trends in health in a population or understanding patients’ experiences of care.
Patient Generated DataData generated directly by patients or their carers including from wearable medical or personal devices, mobile apps, social media, and other internet-based tools. Data can be collected actively (for example, by people entering data on a form) or passively (for example, a smart watch that measures people’s activity level). (NICE, 2022)

In the linked sections we provide guidance specific to each data source, where available. There are also several guides to selecting and evaluating real-world data, some of which are summarised in Table 2. The European Innovative Health Initiative (IHI) project IDERHA (Integration of Heterogeneous Data and Evidence towards Regulatory and HTA Acceptance) has provided a scoping review of existing guidance documents (available here).

Table 2 — Key Sources for Guidance on Real-World Evidence
GuidanceSummary

FDA Real World Evidence Programme		The FDA provided several guidance documents relating to its RWE Program. This includes detailed information on different sources of RWD as well as on standards for regulatory submissions. 

In considering data suitability the FDA emphasises data relevance and reliability in determining data fitness-for-use in regulatory decision-making. Data relevance concerns the availability of key data elements including exposures, outcomes, and covariates. Data reliability concerns accuracy, completeness, provenance, and traceability.
HMA-EMA Data Quality Framework for EU medicines regulation 2022The EMA data quality framework provides general considerations on data quality relevant to regulatory decision-making. The dimensions of data quality include reliability (precision, accuracy and plausibility), extensiveness (completeness and coverage), coherence, timeliness and relevance (availability of relevant data elements).
Duke Margolis Institute for Health Policy 2019, Determining Real-World Data’s Fitness for Use and the Role of Reliability

Duke-Margolis 2018, Characterizing RWD Quality and Relevancy for Regulatory Purposes		Duke Margolis emphasises that data fitness-for-purpose is contextual and depends on the research question. It defines two dimensions for fitness-for-purpose: data relevancy and reliability. Data relevancy concerns the availability of key data elements, representativeness, size, and longitudinality. Data reliability concerns data accrual and data quality, where data quality can be defined by its validity, plausibility, consistency, conformance, and completeness.
TEHDAS 2022, EHDS Data quality frameworkProvides recommendations for ensuring data quality in the European Health Data Space (EHDS) for secondary use. Six prioritised dimensions of data quality were reliability, relevance, timeliness, coherence, coverage and completeness.
Kahn et al. 2016, A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record DataThis framework is used by the US National Patient-Centered Clinical
Research Network (PCORnet), with an additional component, Persistence, and by the Observational Health Data Science and Informatics (OHDSI) network.
National Institute for Health and Care Excellence 2022, NICE Real-world evidence frameworkNICE’s Real-world evidence framework includes guidance on data suitability assessment including an assessment tool (DataSAT).

The NICE framework distinguishes between data provenance and data fitness for purpose. Data provenance focuses on data characteristics, governance, and curation. Data fitness-for-purpose encompasses data quality (accuracy and completeness) and data relevance (availability of key study elements, sample size, follow-up, and representativeness).
CADTH 2023, Guidance for Reporting Real-World EvidenceCADTH does not provide a data quality framework but specifies requirements for data characterisation, data processing, completeness, and validity.

As observed in Table 2, guidance documents differ with respect to terminology and the categorisation of dimensions of data suitability or fitness-for-purpose. However, there is similar coverage in these dimensions across the guidance documents. Castellanos et al. 2024 reviewed several guidance documents and identified the following common dimensions:

  • Relevance – whether the data source contains the data elements needed to answer a research question, whether data is sufficiently large, and representative of the target population.
  • Reliability – depends on the accuracy, completeness, provenance, and timeliness of the data.
Sources of Real World Data

Electronic Health Records

What is it?

EHR data typically consists of structured and unstructured data types. Structured data elements often include patient demographics, medical procedures, diagnoses, and transactional data (such as clinical visits) collected using standardised vocabularies. Unstructured data cannot typically be analysed directly but needs substantial curation. Curation may use automated processes (such as natural language processing), human abstraction, or hybrid approaches. Common types of unstructured data include free-text clinical notes and imaging or pathology reports. In practice, EHR data are frequently pooled from multiple sites and software programs or integrated with non-EHR sources to increase data availability. 

EHR data are often a source for other data sources used for analysis including retrospective chart reviews, patient registries, and audits. They can also be linked to research data from other sources (including trials) to provide additional information on care delivery or clinical outcomes. The FDA considers both medical claims and EHR data to be types of electronic health care data (FDA 2021). We present claims and administrative data separately.

Examples of EHR Data

OpenSAFELY is an open-source software platform for the analysis of primary health electronic health records in England from TPP and EMIS systems. Flatiron Health provide data sources derived from structured and unstructured electronic health record data on clinical outcomes and patient care from cancer providers across the United States.

Why Is It Useful?
  • Opportunity to provide greater depth and detail on patient’s health and the care they receive. This value is increased by the accurate curation of unstructured records within or linked to EHR data.
  • Often cheaper than prospective data collection (including patient registries) and places less burden on healthcare teams collecting and recording data.
What Are Its Key Limitations?
  • Data typically represents what is needed for routine clinical care and some administrative purposes rather than for research questions. This may impact on the availability of relevant data, its granularity, and its accuracy and completeness.
  • Unless systems are well integrated, information on patient care outside of a particular EHR system or provider may be missing (fragmentation of health information).
  • EHR data can be complex with many data types in a wide range of formats (including unstructured records). Substantial curation is commonly needed prior to analysis, and this process can be complex to manage and communicate.
Guidance

Some guidance has been developed to be agnostic to the data source (Table 2). In Table 3 we focus on guidance specific to EHR data.

Table 3 — Key Sources for Guidance on Electronic Health Records
SourceSummary

Assessing Real-World Data from Electronic Health Records for Health Technology Assessment
– The SUITABILITY Checklist: A Good Practices Report of an ISPOR Task Force		* A framework for assessing the suitability of EHR data for use in health technology assessments
* The framework consists of two parts: 1) data delineation and 2) fitness for purpose
* Data delineation concerns the characteristics of the data and its trustworthiness, including data provenance and governance
* Fitness for purpose concerns the reliability and relevance of the data in terms of its ability to answer specific research questions
* The ISPOR SUITABILITY checklist is provided to ensure developers report information to enable a full assessment of data suitability

FDA 2021, Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products		* Guidance issued as part of FDA’s RWE Program
* The guidance covers the selection of data sources to ensure relevance to the research question, the development and validation of definitions for study design elements, and data provenance and quality during data accrual, data curation, and into the final analysis dataset
Sources of Real World Data

Patient Registries

What is it?

Patient registries are organised systems that collect uniform data (clinical and other) to identify specified outcomes for a population defined by a particular disease, condition or exposure (AHRQ 2020). Registries can serve several purposes including research, clinical care or policy. They are sometimes constructed in part from paper or electronic health record systems but often involve additional data collection.

Examples of Patient Registries

The European Society for Blood and Marrow Transplantation (EBMT) contains clinical data on more than 700,000 patients since 1974 who have received a haematopoietic cell transplantation (HCT) procedure and over 4,000 patients who received CAR-T cell therapy.

The European Cystic Fibrosis Society Registry (ECFS)  is used to collect data on cystic fibrosis and its treatment, both to improve treatment and provide data for epidemiological research.

Why Is It Useful?
  • Data models can be designed to directly inform specific research questions, improving the relevance of the data, and quality control processes can ensure its reliability
  • Prospective data collection may allow for the collection of data elements that are commonly absent from routinely collected RWD sources such as patient-reported outcome measures
  • Greater opportunity for direct patient involvement in data collection
What Are The Key Challenges?
  • Recruiting a sufficient number of patients representative of the target population and ensuring high-quality and complete data over follow-up 
  • Data may be collected for specific purposes that lead to a narrower patient population or set of variables than needed for other research questions
  • Sustained funding to maintain high-quality data collection over time (integration with EHR systems can help)
  • Ensuring coordination between ongoing initiatives at national and international levels
  • Limited data interoperability, data sharing and transparency of some registries impacting usability and trust
Guidance

Some guidance has been developed to be agnostic to the data source (Table 2). In Table 4 we focus on guidance specific to registries. 

Table 4 — Key Sources for Guidance on Patient Registries
SourceSummary
FDA 2023, Real-World Data: Assessing Registries To Support Regulatory Decision-Making for Drug and Biological Products* Guidance issued as part of FDA’s RWE Program
* Provides information on assessing a registry’s fitness-for-use for decision-making
* Focuses on assessment of data relevance and reliability
* Provides considerations when linking a registry with other data sources

EMA 2021, Guideline on registry-based studies – Scientific guideline		* Addresses the methodological, regulatory, and operational aspects involved in using registry-based studies to support regulatory decision-making.
* Provides guidance on data collection, data quality management and data analysis
IQWIG 2020, [A19-43] Development of scientific concepts for the generation of routine practice data and their analysis for the benefit assessment of drugs according to §35a Social Code Book V – rapid report* Specifies criteria for data quality and reporting from patient registries and guidance on analysis of the data, focused on quantifying the added benefit of a new drug
EUnetHTA 2019, REQueST® Tool and its Vision paper* Developed through collaboration with European HTA bodies through EUnetHTA Joint Action 3
* The Registry Evaluation and Quality Standards Tool (REQueST) aims to support HTA organisations and other actors in guiding and evaluating registries for effective usage in HTA.
* The purpose is to highlight areas of a registry that need improvement to maximise the quality of its data and ensure that those data can be used for HTA and regulatory purposes
Agency for Healthcare Research and Quality (AHRQ) 2020, Registries for Evaluating Patient Outcomes: A User’s Guide: 4th Edition* Provides guidance on registry planning, design, operations, and analysis, including on the integration of existing data sources and use of common data elements and standardised outcome measures
Sources of Real World Data

Administrative & Claims Data

What is it?

Data that is collected for administrative purposes including healthcare planning and financing. Claims data are administrative data on the use of healthcare services often collected in insurance-based systems.

Examples of Administrative & Claims Data

The Hospital Episode Statistics (HES) data warehouse in England provides information on diagnoses and procedures received in NHS hospitals during emergency department visits, outpatient appointments or admitted care. It is primarily used for the reimbursement of hospitals for services provided and other operational activities.

The Centers for Medicare & Medicaid Services data contains data on individuals receiving Medicare services derived from reimbursement information or payment of bills.

Why Is It Useful?
  • Data is highly structured and no additional burden is placed on patients or healthcare teams
  • Data on healthcare utilisation and costs can be useful for pricing and reimbursement decisions 
  • Availability and lower cost of access
What Are The Key Challenges
  • Data is collected for billing and other operational purposes rather than research. Data may not be collected with sufficient granularity or reliability and important patient demographic and clinical data (e.g., tests, signs, symptoms, vital status) may not be collected. 
  • Potential lag in data availability for research
  • In some healthcare systems, follow-up of patients can be limited as patients switch insurance providers often.
Guidance

Some guidance has been developed to be agnostic to the data source (Table 2) Here we focus on guidance specific to administrative and claims data. 

Table 5 — Key Sources for Administrative & Claims Data
SourceSummary

FDA 2021, Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products		* Guidance issued as part of FDA’s RWE Program applies also to claims data
* The guidance covers the selection of data sources to ensure relevance to the research question, the development and validation of definitions for study design elements, and data provenance and quality during data accrual, data curation, and into the final analysis dataset

Kahn et al. 2016, A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use
of Electronic Health Record Data		* This framework is used by the US National Patient-Centered Clinical Research Network (PCORnet), with an additional component, Persistence, and by the Observational Health Data Science and Informatics (OHDSI) network.
Generating Real-world Evidence

Overview

RWE is commonly defined as evidence generated from the analysis of real-world data. 

The type of evidence generated from RWD varies widely. It can include characterizing patients, conditions, or care delivery; generating economic models; and/or estimating comparative effectiveness or cost-effectiveness.  The appropriate study design depends on the research question and its context. 

In this section, we focus on study designs for estimating the comparative effects of interventions using RWD. We consider experimental, observational, and mixed designs. Short summaries of each method are described in Table 6 below, with more information in the links for the key study designs. 

Information on the key challenges in estimating causal effects using observational study and analytical methods for addressing bias are discussed in a separate section. We also discuss ways of assessing the validity of RWE studies. 

Table 6 — Study Designs Applicable to Use of RWD for Comparative Effect Estimation
Common Study Designs
Cohort StudyA type of longitudinal observational study which collects data on a defined group of people over time. Cohort studies can be used to estimate the causal impact of exposures (including medical interventions) on clinical or other outcomes.
External Control Arm (ECA) for Clinical TrialA special case of a cohort study in which the data for people with different exposures comes from different data sources. Typically, data on a new treatment from a clinical trial is compared to data from an external source, either a previous trial or real-world data, to form an indirect treatment comparison.
Pragmatic Randomised Controlled TrialA pragmatic trial aims to measure the relative effectiveness of treatments in real-world clinical practice retaining the benefits of randomisation. Pragmatic randomised controlled trials may use real-world data in several ways including for recruitment or patient follow-up.
Other Observational Designs
Quasi-Experimental StudiesAim to estimate causal effects of exposures using external variation in exposure across people or over time that is otherwise unrelated to the outcome. Examples of quasi-experimental methods include instrumental variable analysis, regression discontinuity, interrupted time series and difference-in-difference estimation.
Case-ControlA study that examines associations between outcomes and prior exposures by comparing people with an outcome of interest to those without the outcome. These generally do not allow for  quantification of relative risk and are not used to estimate the relative effectiveness of interventions.
Cross-SectionalData are collected from a population or a representative subset of a population at one specific point in time (or over a short  period) to examine associations between health status and exposures (use of interventions). The cross-sectional relationships revealed generally are not robust evidence of relative effectiveness, which requires longitudinal data.
Self-ControlledSelf-controlled, or ‘within-subject’, designs make use of variation in exposure status within individuals over time. These include case-crossover, self-controlled case series, and variants of these designs. They are most appropriate for transient exposures with acute-onset events (Hallas and Pottegard 2014).
Case Report / Case SeriesA detailed report on a smaller number of patients, typically describing symptoms an unusual or new occurrence, including outcomes after a treatment.
Other Experimental Designs
Population Enrichment RCTIncludes patients with characteristics typically under-represented or excluded from RCTs. Predictive modelling techniques may be applied to data generated from these studies to facilitate the estimation of relative effectiveness in a real-world population.
Cohort Multiple RCT (cmRCT) (also known as trials within cohorts)A type of pragmatic RCT that uses a large cohort of patients as a source of participants for a variety of RCTs, providing a more generalisable study sample.
Comprehensive Cohort Study (CCS)A type of pragmatic RCT that includes participants who do not consent to be randomised to the treatment group. This facilitates reduction in selection bias and improves the generalisability of study results.
Cluster RCTCluster RCTs randomise groups or clusters rather than individual participants as in traditional RCTs. This can reduce the possibility of contamination of the comparator (usual care) group.
Non-Randomised Controlled TrialAny experimental study in which patients are allocated to different treatments using a method other than randomisation, such as clinician or patient preference.
Generating Real-world Evidence

Cohort Studies

What is it?

Cohort studies are a type of longitudinal observational study which collects data on defined groups of people over time. Cohort studies can be used to estimate the causal impact of exposures (including medical interventions) on clinical or other outcomes. They are common in pharmacoepidemiology to study the use and effects of medical products in routine care. There is growing interest in their use to extend evidence from RCTs (for instance, to populations underrepresented or excluded from trials) or to answer questions where RCTs are lacking. Such studies are most relevant for studying the use or effects of treatments already used in routine practice, but may limited for initial evaluations of interventions since, by definition, limited real-world data will have accrued.

Several recent guidance documents have advocated the use of target trial designs whereby researchers articulate the RCT they would like to have performed and attempt to replicate this as closely as possible using observational data. This approach can be combined with several different analytical methods including the use of propensity scores. See section on Addressing Bias for more information.

Advantages & Challenges

Advantages of cohort studies over other observational designs include the ability to model temporal relationships between exposures and outcomes (compared to cross-sectional designs) and greater consistency of data collection for individuals with different exposures (compared to external control arms). 

As with other observational designs, there remain significant challenges in addressing sources of bias including selection bias, confounding, and information biases from data mismeasurement or inaccuracy.

What Do We Know About Their Validity?

Numerous emulation studies have compared estimates of comparative effects from randomised controlled trials to observational cohort studies. We note two larger studies, as follows:

  • Cochrane Library performed a meta-epidemiological study comparing studies addressing the same question using both randomised and observational designs (Toews et al. 2024). The study included 47 systematic reviews involving 2869 RCTs and 3924 observational studies. Overall, results suggested little difference in effect estimates obtained from RCTs versus those from observational studies (ratio of ratios 1.08, 95% confidence interval 1.01 to 1.15). 
  • RCT-Duplicate emulated the design of 32 RCTs of medications across different indications using US claims data (Wang et al. 2023). Overall, they found reasonable overall agreement between the results of RCTs and observational studies (person correlation 0.82, 95% CI, 0.64 to 0.91). This correlation was greater where the ability to emulate trials designs was greater (0.93, 95% CI, 0.79 to 0.97).

Several important guidance documents have been published to support the generation of evidence using cohort studies including:

Generating Real-world Evidence

External Control Arms

Overview

External control arm (ECA) studies are a special case of cohort studies in which the data for people with different exposures comes from different data sources. Typically, data on a new treatment from a clinical trial is compared to data from an external source, either a previous trial or real-world data. ECAs are most seen in initial evaluations of new treatments when there is no randomised controlled trial in which the new treatment is compared against the comparator of interest. External control arms are most common when only single-arm trials have been conducted, when the control arm in a clinical trial is not the local standard of care, or for contextualisation.

Advantages & Challenges

ECAs allow comparison against relevant treatments when ethical or feasibility considerations preclude direct comparison within RCTs. In addition to the challenges of cohort studies stated above, additional challenges can arise due to substantial differences between RCT and RWD data. In particular, data collection processes and study design elements may limit the comparability of patients and/or outcomes (for instance, cancer progression may be defined differently in trials vs routine care) and impair emulation of the trial design.

Validity of Studies

While several emulation studies have been published, there have been fewer systematic high-quality studies across different contexts than for traditional cohort studies.

Guidance

A growing body of guidance documents has been published to support the generation of evidence using external control arms including:

What Is It?

Pragmatic RCTs aim to estimate the relative effectiveness of treatment strategies in real-world clinical practice and generate ‘real-world evidence’. 

Pragmatic and explanatory trials (which measure efficacy under ideal conditions, such as typical phase 3 RCTs) represent the ends of a continuum rather than distinct entities (Thorpe et al, 2009). A study may contain elements from both approaches. The design choices that can be made towards a more pragmatic trial design can be related to four domains: the study population; the setting of the trial; the operationalisation of the intervention and choice of comparator treatment; and the outcome measure. General issues of data management and monitoring also need to be considered, because these can influence routine clinical practice and therefore the generalisability of the trial results.

Why Is It Useful?
  • Randomisation: Randomisation of patients to treatments increases the validity of the study findings by, on average, balancing baseline characteristics of patients receiving different interventions. 
  • Generalisability: Provide direct evidence on effectiveness in the population and setting of interest.
  • Efficiency: More efficient recruitment and follow-up, lower burden on patients and healthcare professionals.
What Are The Limitations?

Operational Challenges: Pragmatic trial designs may lead to different and unanticipated operational challenges compared with typical phase 3 (explanatory) trials. For example, the participation of real-world prescribers may require study sites to be moved from specialised trial centres to medical departments or primary care settings, which may not be fully equipped or experienced in conducting a clinical trial.

Complex Interplay Between Study Design and Operational Challenges: Designers of pragmatic trials need to be aware of the consequences of their study design choices and consider the implications of these choices on the generalisability of results to routine practice, validity and precision of results, their acceptability to stakeholders and operational feasibility of the study.

Addressing Bias

Target Trial Design

Recent methodological guidance has referenced the target trial approach to study design. Target trial emulation is a framework for designing and analysing observational studies that aim to estimate the causal effect of interventions and can be combined with any observational design (Fu et al. 2024). In brief, the target trial design involves articulating the trial that developers would ideally perform and describing how RWD will be used in this process across several design elements, including: 

  • Eligibility criteria 
  • Treatment strategies
  • Treatment assignment
  • Outcomes
  • Causal estimand
  • Start and end of follow-up
  • Statistical analysis

There are several potential advantages to using the target trial design including:

  • Improving transparency around study design and trade-offs made when emulating a hypothetical trial using RWD
  • Improving the quality of studies by improving design and avoiding common analytical errors

Other complementary approaches to improving study design and its articulation include study design diagrams for reporting on study design and the use of directed acyclic graphs (DAGS) for identification and visualisation of causal effects and confounders.

Addressing Bias

Statistical Methods for Addressing Confounding

There are many statistical methods for addressing confounding. Below we summarise common methods used for estimating the comparative effects of interventions. In addition to the below, quantitative bias analysis is also recommended for examination of the impact of unmeasured confounding and/or to explicitly model its impact on estimation.

Table 7 — Common Methods for Estimating Comparative Effects of Real-world Interventions
MethodSummary

Adjustment Using Regression Models		Regression models can be used to estimate the difference in outcomes between groups controlling for covariates. The type of regression model will depend on the outcome and its distribution. Common methods include the cox-proportional hazards model for time-to-event outcomes (e.g., overall survival), logistic regression for binary outcomes (e.g., myocardial infarction in 1 year), or linear regression for continuous outcomes (e.g., blood pressure).
Propensity Score MethodsPropensity scores provide the probability of an individual (or other study unit) belonging to a specific group (e.g., received a specific drug) based on their characteristics. They are typically estimated using logistic regression models. 

Propensity scores can be used to estimate treatment effects in various ways. A selection of the most common methods is noted below (see Ali et al. 2019 for a fuller description).
Inverse Probability Weighting (IPW)Propensity scores are used to weight observations in each group such that the distribution of baseline covariates is similar across treatment groups. Mean weighted outcomes across groups are then compared.
StratificationObservations are divided into strata according to their propensity scores. Differences in outcomes within each stratum are estimated and then combined across strata using a simple or weighted average.
MatchingIndividuals with similar propensity scores are identified and matched. The treatment effect is the average difference between matched pairs. There are numerous algorithms to support matching including caliper, nearest neighbour, and kernel matching. Regression models can also be applied to matched data to address residual confounding.

Exact matching based on a small number of patient characteristics (e.g., age and sex) can also be done without use of propensity scores.
Doubly Robust MethodsUses propensity scores to weight observations within a regression model. It is known as doubly robust because it can provide consistent estimates of treatment effects if either the outcome model or the propensity score model is correctly specified.
Disease Risk ScoresDisease risk scores estimate the probability of observing a specific outcome based on patient characteristics. They can be used in regression models or for matching.
Methods for Addressing Time Varying ConfoundingCommon methods include marginal structural models and G-estimation.
Methods for Addressing Unknown or Unmeasured Confounding Instrumental Variable MethodsCommon methods include instrumental variable regression, difference-in-difference, panel data methods. These are sometimes called quasi-experimental methods.
Addressing Bias

Assessing Risk of Bias

The Comparative Effectiveness Research Special Interest Group of the International Society for Pharmacoepidemiology (ISPE) presented a systematic literature review of non-randomised study assessment tools (D’Andrea et al. 2021). They found no single tool that covered all risks of bias. GRACE and ROBINS-I were considered the most complete tools. 

Some HTA bodies such as NICE have recommended the use of ROBINS-I to assess the validity of non-randomised studies. Other decision-making bodies use their own criteria to assess validity.

Research Governance

Overview

Most recent guidance from regulatory and HTA bodies emphasise the importance of following best practices in research governance ensuring transparency and integrity in all stages of the study from design through to conduct and reporting. Below we identify and summarise key guidance documents to support implementation of these best practices. We present tools which involved regulatory or HTA bodies or have otherwise been endorsed by them. These should be used in addition to formal guidance issued by regulatory and HTA bodies.

Table 8 — Guidance Documents to Implement Best-Practices in Study Designs
Guidance/ToolSummary
PRINCIPLEDProvides step-by-step guide following best practices for designing and analysing non-interventional (RWE) studies that ensure transparent, reliable, and reproducible evidence.
START-RWEA structured template for planning and reporting on the implementation of RWE studies.
ISPOR RWE Transparency Initiative: Recommendations and RoadmapRecommendations for processes to ensure transparency and trustworthy real-world evidence. The report recommends publication of pre-specified protocols on publicly accessible platforms.
REQueST ToolA tool to help decide whether to use registry data for regulatory or HTA purposes.
CADTH Guidance for Reporting RWE
A reporting checklist for RWE studies informing decisions in Canada.
HARPERA good practices report from a joint ISPOR/ISPE taskforce, providing a template for protocols of RWE studies of treatment effects.
STROBEA checklist for reporting observational studies
ESMO Guidance for Reporting Oncology real-World Evidence (ESMO-GROW)Guidance and checklist for reporting real-world evidence studies in oncology
Good Practices for Real-World Data Studies of Treatment and/or Comparative EffectivenessAn ISPOR task force report providing recommendations for good procedural practice for studies that test a specific hypothesis in a specific population
Guidance Documents Repository

Regulatory Agency Publications on Real-World Data/Evidence

European Medicines Agency (EMA)
US Food and Drug Administration (FDA)
MHRA
China NMPA Center for Drug Evaluation
Japan’s Pharmaceutical and Medical Devices Agency (PMDA)
Health Canada
Australian Therapeutic Goods Administration (TGA)
Swiss Medic
Guidance Documents Repository

Health Technology Assessment Publications on Real-World Data/Evidence

NICE
Canada’s Drug Agency – L’Agence des Médicaments du Canada (CDA-AMC)
Haute Authorité de Santé (HAS)
Guidance Documents Repository

Professional Associations

The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH)
Council for International Organization of Medical Sciences (CIOMS)
Scroll to Top