Risk of bias assessment: the Cochrane risk of bias tool (RoB 2), ROBINS-I, AMSTAR 2 and Newcastle-Ottawa

Risk of bias is the likelihood that a study gets the wrong answer because of flaws in its design, conduct or analysis, not because of a real effect. A risk of bias assessment means judging how far you can trust the findings of each study included in a review, and the Cochrane risk of bias tool (RoB 2) is the standard way to do it for randomized trials. Why does it matter so much? A systematic review that pools badly biased studies does not produce reliable evidence. It only adds up the error. Two poorly conducted studies do not cancel each other out; combining them gives the same bias a false look of precision. So before you synthesize anything, you have to appraise the methodological quality of each study with a validated tool chosen for its design.

The standard tools for that job are RoB 2 for randomized trials, ROBINS-I for non-randomized studies of interventions, the Newcastle-Ottawa Scale for observational studies and AMSTAR 2 for published systematic reviews. At the end of the process, GRADE rates the overall certainty of the evidence.

What risk of bias is (and why it matters)

Bias is any systematic error that pushes a study's results away from the true value. Random error shrinks as samples get larger. Bias does not. Adding participants will not remove it, because it is built into the way the study was run. A trial in which patients know which treatment they are getting can overestimate the benefit, and no sample size will fix that.

In a systematic review, assessing risk of bias does three jobs:

• It tells you how much weight to give each study. A study at low risk of bias counts for more in the interpretation than one at high risk.
• It helps explain heterogeneity. When results disagree, part of the disagreement may come from differences in methodological quality, not from the true effect.
• It justifies your confidence in the conclusions. It is the foundation for rating the overall certainty of the evidence with GRADE, the system behind evidence-based medicine.

Three terms often get mixed up. Risk of bias is about how well the study was done (its internal validity). Methodological quality is a broader term that usually covers risk of bias plus other aspects of reporting. And critical appraisal is the general process of reading a paper with a critical eye; risk of bias assessment is the formal, reproducible part of it.

Types of bias (selection, performance, detection, publication)

Modern tools are organized by domains, but it helps to know the main families of bias, because each tool covers them differently.

• Selection bias. It appears when the groups being compared differ at baseline in something other than the intervention. In a trial it is controlled by proper, concealed random allocation (nobody can predict or manipulate which group each participant ends up in). Selection bias is the classic threat in observational studies, where groups are not formed at random.
• Performance bias. It comes from differences in the care the groups receive beyond the intervention under study, usually because participants or staff know which treatment is being given. Blinding (masking) reduces it.
• Detection bias. It happens when the person measuring the outcome knows the allocation and that shapes how they record it, especially for subjective outcomes. Blinding the outcome assessor is the main defense.
• Attrition bias. It stems from loss to follow-up and incomplete data when those losses are not spread evenly across groups.
• Reporting bias. It occurs when only some outcomes are published, usually the significant ones, and the rest are hidden.
• Publication bias. It works at the level of the whole body of studies, not a single one. Studies with positive results get published more often and sooner than negative or inconclusive ones. That distorts any meta-analysis built on the published record. It is explored with methods such as the funnel plot and assessed at the synthesis stage, not with single-study tools.

Each study type has its own instrument, and using the wrong one invalidates the assessment. The tools below split the work by design.

RoB 2, the Cochrane risk of bias tool (randomized trials)

RoB 2 (Risk of Bias 2) is Cochrane's tool for assessing risk of bias in randomized controlled trials (RCTs). It was published in 2019 as an update of the original 2008 version. If your review includes randomized trials, this is the instrument to use.

RoB 2 does not produce a numeric score. Instead it structures the judgment into five domains, and for each one you answer a set of signalling questions that lead to a judgment guided by an algorithm. The five domains are:

1. Bias arising from the randomization process.
2. Bias due to deviations from intended interventions.
3. Bias due to missing outcome data.
4. Bias in measurement of the outcome.
5. Bias in selection of the reported result.

Each domain is judged as "Low risk of bias", "Some concerns" or "High risk of bias". The overall judgment follows conservative logic. A single domain at high risk is enough to put the whole trial at high risk. RoB 2 is also applied to a specific outcome rather than to the study as a whole, because the same trial can be at low risk for an objective outcome (mortality) and at high risk for a subjective one (patient-reported pain).

ROBINS-I (non-randomized studies)

When the included studies are not randomized but still estimate the effect of an intervention (for example, cohort studies comparing two treatments assigned in routine practice), the right tool is ROBINS-I (Risk Of Bias In Non-randomised Studies of Interventions), also from Cochrane.

ROBINS-I starts from the "target trial", the ideal randomized trial that would answer the same question. The assessment measures how far the real study strays from it. That is why it includes domains an RCT would not need, above all confounding, the main problem of non-randomized studies. Its seven domains are:

1. Bias due to confounding.
2. Bias in selection of participants into the study.
3. Bias in classification of interventions.
4. Bias due to deviations from intended interventions.
5. Bias due to missing data.
6. Bias in measurement of outcomes.
7. Bias in selection of the reported result.

The judgment categories go one step further than in RoB 2: low, moderate, serious, critical, or no information. A well-conducted observational study rarely reaches "low risk" in ROBINS-I, because without randomization the door to residual confounding is always open. That is not a flaw in the tool. It reflects a real limitation of the design.

Newcastle-Ottawa Scale (observational studies)

The Newcastle-Ottawa Scale (NOS) is one of the most widely used tools for rating the quality of observational studies, specifically cohort and case-control studies. Unlike RoB 2 and ROBINS-I, the Newcastle-Ottawa Scale does use a star system. It awards points in three blocks (selection of the study groups, comparability of the groups, and ascertainment of the outcome or the exposure), with a maximum of around nine stars depending on the version.

Its main strengths are simplicity and speed, which explain much of its popularity. Its limitations are just as well known. The scoring can hide serious problems (a study with a critical flaw in a single item can keep collecting stars on the others), and the cut-offs for calling a study "good" or "poor" are not well standardized across reviews. That is why, in reviews that aim for the highest methodological rigor, many teams prefer ROBINS-I for observational studies of interventions and keep the Newcastle-Ottawa Scale for exposure-outcome studies or for when the protocol specifies it. Either way, state in advance which tool you will use and why.

In revisia, the risk of bias assessment of your included studies happens in the same workflow and comes out ready for the manuscript. Create your free account.

AMSTAR 2 (quality of systematic reviews)

The tools above appraise primary studies. AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews, version 2, from 2017) moves up a level and appraises the methodological quality of systematic reviews themselves. You use it when your objects of study are not trials or cohorts but other reviews (for example in an umbrella review), or when you want to judge a published review critically before trusting its conclusions.

AMSTAR 2 has 16 items. It does not produce a total score. Instead it identifies seven critical domains (such as registering a protocol in advance, the adequacy of the literature search, justification for excluding studies, and the risk of bias assessment of the included studies). The number and weight of the weaknesses in those domains set four levels of confidence in the review: high, moderate, low or critically low. A single critical flaw is usually enough to drop confidence to low.

One of its items checks exactly whether the review assessed the risk of bias of its studies. Without that step, no review can hope for a high rating. If you want to go deeper into how the pooled result of a review is interpreted, read our guide on meta-analysis and how to read a forest plot, where publication bias comes back through the funnel plot.

ROBIS and other specific tools

For systematic reviews there is also ROBIS (Risk of Bias in Systematic Reviews), a tool focused specifically on risk of bias in the review process. It is organized into four domains (study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings). ROBIS and AMSTAR 2 complement each other. AMSTAR 2 looks at overall methodological quality and ROBIS at specific risk of bias. Despite the name, ROBIS has nothing to do with similarly named commercial brands.

For particular study types or outcomes there are further tools (for example, the JBI critical appraisal checklists for different designs, or QUADAS-2 for diagnostic accuracy studies). The rule does not change. Pick the tool validated for your design instead of forcing a generic one.

Which tool to use for each study design

A quick reference.

GRADE and levels of evidence

The tools above appraise studies or reviews one at a time. GRADE (Grading of Recommendations, Assessment, Development and Evaluations) works on a different plane. It rates the overall certainty of the evidence for each outcome, that is, how confident we can be that the estimated effect is close to the true one. It is not a single-study risk of bias tool but the final step that combines risk of bias with other factors.

GRADE rates the certainty of the evidence on four levels: high, moderate, low and very low. The starting point depends on the design (randomized trials start at high certainty and observational studies at low), and from there certainty can be rated down for five reasons:

1. Risk of bias (this is where the results of RoB 2, ROBINS-I and similar tools come in).
2. Inconsistency (unexplained heterogeneity between studies).
3. Indirectness (the population, intervention or outcome do not fully match the question).
4. Imprecision (wide confidence intervals, few events).
5. Publication bias (suspicion that unpublished studies are missing).

For observational studies, certainty can also be rated up for factors such as a large magnitude of effect or a dose-response gradient. The resulting level of evidence then underpins the strength of clinical recommendations in evidence-based medicine. That is why GRADE closes the process. If you do not appraise each study well, you cannot rate the body of evidence well.

How to report it under PRISMA

Assessing risk of bias is worthless if it is not documented in a transparent, reproducible way. The PRISMA 2020 statement devotes several items to it. You have to state which tool you used, how many reviewers assessed each study independently and how disagreements were resolved, and present the results of the assessment (usually with a traffic light plot by domain for RoB 2 and ROBINS-I, or a star table for the Newcastle-Ottawa Scale). Publication bias is reported separately, usually with a funnel plot and, where appropriate, an associated statistical test.

For the report to follow the guideline and get past peer reviewers and journals, stick to the official structure. You can see how the risk of bias assessment fits into the flow and the full checklist in our guide to the PRISMA 2020 flow diagram and its template.

Risk of bias assessment is one of the core steps of the method. To place it within the whole process, from the question to the synthesis, go back to the main guide on how to do a systematic review step by step.

In revisia you assess risk of bias inside the same workflow as the rest of your review. Create your free account.

Frequently asked questions