What kappa measures
Kappa measures how often two people who classify the same things agree, after taking out the agreement they would reach by chance. In a systematic review it is used mainly for dual screening: two reviewers read the same titles and abstracts and decide independently whether each record goes in.
It is κ = (Po − Pe) / (1 − Pe). Po is the share of records the two agree on. Pe is the share you would expect if each decided at random with their own tendency to include. A kappa of 1 is total agreement; 0 is what chance gives; below 0, less than chance.
How to interpret it
The Landis and Koch (1977) scale is the one almost every review quotes:
| κ | Agreement |
|---|---|
| < 0 | Poor |
| 0.00 – 0.20 | Slight |
| 0.21 – 0.40 | Fair |
| 0.41 – 0.60 | Moderate |
| 0.61 – 0.80 | Substantial |
| 0.81 – 1.00 | Almost perfect |
Landis and Koch called their cut-offs arbitrary. Health research often asks for more: McHugh (2012) treats anything under 0.60 as inadequate. Give the number and its interval, not only the label.
Worked example
Two reviewers screen 500 records on title and abstract. Both include 20 and both exclude 465; reviewer A alone includes 8 and reviewer B alone includes 7. It is the example the calculator opens with.
- Observed agreement: Po = (20 + 465) / 500 = 0.970.
- Reviewer A includes 28 of 500 (0.056) and reviewer B 27 of 500 (0.054).
- Chance agreement: Pe = 0.056 × 0.054 + 0.944 × 0.946 = 0.896.
- Kappa: κ = (0.970 − 0.896) / (1 − 0.896) = 0.71, 95% CI 0.57 to 0.85.
97% agreement ends up as a kappa of 0.71, substantial agreement: almost everything is excluded and chance already explained 89.6%. That is the paradox explained below.
Why screening kappa comes out low
In screening, excluding 90% or more of the records is normal. When almost everything is «exclude», chance already explains most of the agreement, and a couple of disagreements among the few includes pull kappa down. Two reviewers can agree on 97% of records and get a kappa of 0.6.
This is the kappa paradox (Feinstein and Cicchetti, 1990). The calculator flags it when it shows up and gives three numbers that help explain it: the prevalence index (how much one category outweighs the other), the maximum kappa the margins allow, and PABAK, the prevalence- and bias-adjusted kappa of Byrt, Bishop and Carlin (1993).
Which kappa to use
- Cohen’s kappa. Two raters, unordered categories: include or exclude, yes or no, study design. The usual one for screening.
- Weighted kappa. Two raters, ordered categories: low, some concerns or high risk; a 1 to 5 scale. With quadratic weights a two-step disagreement counts four times a one-step one; with linear, twice. Quadratic weighted kappa is close to equivalent to the intraclass correlation coefficient (Fleiss and Cohen, 1973).
- Fleiss’ kappa. Three or more raters who all classify the same records. With two raters it gives Scott’s pi, close to Cohen’s kappa but not the same.
When kappa is the wrong measure
- Continuous scores. For continuous ratings (a 0 to 100 scale, a time) use the intraclass correlation coefficient (ICC), not kappa.
- Missing ratings or different raters per record. If not every rater rates every record, Krippendorff’s alpha allows it; Fleiss’ kappa needs the same number of raters on each record.
- One very rare category. When almost everything falls in one category, Gwet’s AC1 suffers less from the paradox than kappa. If you use it, report kappa too: it is what journals expect.
How to report it in your review
PRISMA 2020 asks how many reviewers screened each record and whether they worked independently (item 8); it does not require kappa, but many journals and examiners expect it. When you give it, include the type of kappa, its value with the confidence interval, the number of records and the stage it was measured at (titles and abstracts or full text).
Cite the method you use: Cohen J. Educ Psychol Meas 1960;20:37-46 for Cohen’s kappa; Cohen J. Psychol Bull 1968;70:213-20 for weighted kappa; Fleiss JL. Psychol Bull 1971;76:378-82 for Fleiss’ kappa; Landis JR, Koch GG. Biometrics 1977;33:159-74 for the scale.
Frequently asked questions
What is a good kappa?
On the Landis and Koch scale, 0.61 to 0.80 is substantial agreement and 0.81 to 1 almost perfect. For systematic review screening, a kappa above 0.60 usually satisfies journals and examiners. If yours is lower, look at the disagreements: they almost always point to an ambiguous eligibility criterion worth rewriting before you go on.
How is the confidence interval calculated?
For Cohen’s kappa, weighted or not, with the large-sample standard error of Fleiss, Cohen and Everitt (1969): the one the cohen.kappa function in the R package psych uses. Fleiss’ kappa has no closed-form interval the packages agree on, so its interval comes from 2,000 bootstrap resamples of the records, with a fixed seed so the same data always gives the same interval.
Does it match SPSS or R?
Yes. Kappa, its standard error and the test match kappa2 and kappam.fleiss in the R package irr and cohen.kappa in psych, checked to ten decimal places. SPSS gives the same Cohen’s kappa.
Can I paste ratings straight from Excel?
Yes. Select one column per rater in Excel or Google Sheets, copy, and paste into «Paste decisions». Any labels work (Include/Exclude, Yes/No, 1/0) and case does not matter. A first row with the raters’ names is detected on its own. Rows with an empty cell are set aside and the calculator tells you how many.
Why do I get 95% agreement and a kappa of 0.6?
Because almost every record falls in the same category, as in screening, where the great majority is excluded. With that split chance already predicts a lot of agreement, and kappa only measures what goes beyond chance. This is the kappa paradox. Report kappa together with the observed agreement and, if you like, PABAK, which the calculator gives whenever kappa is unweighted.
Does my data leave the browser?
No. Everything is computed and drawn on your device, and what you type is kept only in your browser’s local storage, so a reload does not lose it. No account needed.
Why does the figure carry a watermark, and how do I remove it?
The figure you download as PNG or SVG carries a revisia mark at the bottom right, outside the table. Sign in to revisia with a verified email and it comes out clean, in the preview and in both formats. The account is free and there is no paid version of this tool. The methods sentence is always copied without a mark.